[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:50 dist-test-slave-mlmz krb5kdc[23735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:50 dist-test-slave-mlmz krb5kdc[23735](info): set up 2 sockets
Sep 29 16:30:50 dist-test-slave-mlmz krb5kdc[23735](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 16:30:51.505533 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:30:51 dist-test-slave-mlmz krb5kdc[23735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:30:51.622831 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:32975
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:32975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:51.730469 23748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:51.730696 23748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:51.730734 23748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:51.734576 23748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:51.734625 23748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:51.734644 23748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:51.734663 23748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:51.734678 23748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:51.739641 23748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:32975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:32975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:51.740517 23748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:51.741463 23748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:51.750156 23754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:51.750169 23756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:51.750197 23753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:51.750770 23748 server_base.cc:1034] running on GCE node
I20260929 16:30:51.751322 23748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:51.752408 23748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:51.753623 23748 hybrid_clock.cc:648] HybridClock initialized: now 1790699451753610 us; error 36 us; skew 500 ppm
Sep 29 16:30:51 dist-test-slave-mlmz krb5kdc[23735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699451, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:51.756296 23748 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:30:51.757081 23748 webserver.cc:460] Webserver started at http://127.23.44.62:33375/ using document root <none> and password file <none>
I20260929 16:30:51.757733 23748 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:51.757823 23748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:51.758038 23748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:51.760093 23748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "680fcbc84bba4e7a95eff244c429aa35"
format_stamp: "Formatted at 2026-09-29 16:30:51 on dist-test-slave-mlmz"
server_key: "a8aa9fe60b0306a58179d3c9878085a5"
server_key_iv: "56f7ed0f6bd0227ef034fa0b2c84b7bb"
server_key_version: "encryptionkey@0"
I20260929 16:30:51.760710 23748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "680fcbc84bba4e7a95eff244c429aa35"
format_stamp: "Formatted at 2026-09-29 16:30:51 on dist-test-slave-mlmz"
server_key: "a8aa9fe60b0306a58179d3c9878085a5"
server_key_iv: "56f7ed0f6bd0227ef034fa0b2c84b7bb"
server_key_version: "encryptionkey@0"
I20260929 16:30:51.764189 23748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:30:51.766476 23763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:51.767805 23748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:30:51.767947 23748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "680fcbc84bba4e7a95eff244c429aa35"
format_stamp: "Formatted at 2026-09-29 16:30:51 on dist-test-slave-mlmz"
server_key: "a8aa9fe60b0306a58179d3c9878085a5"
server_key_iv: "56f7ed0f6bd0227ef034fa0b2c84b7bb"
server_key_version: "encryptionkey@0"
I20260929 16:30:51.768057 23748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:51.777865 23748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:51.780424 23748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:51.780606 23748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:51.787110 23748 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:32975
I20260929 16:30:51.787122 23815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:32975 every 8 connection(s)
I20260929 16:30:51.787966 23748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:51.790598 23816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:51.795568 23816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35: Bootstrap starting.
I20260929 16:30:51.797915 23816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:51.798520 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 23748
I20260929 16:30:51.798683 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:30:51.798794 23816 log.cc:824] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:51.800151 23728 external_mini_cluster.cc:1428] Setting key 8280b5cc21292c8fab53f9e3adaaaf8f
I20260929 16:30:51.800887 23816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35: No bootstrap required, opened a new log
I20260929 16:30:51.805181 23816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "680fcbc84bba4e7a95eff244c429aa35" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 32975 } }
I20260929 16:30:51.805433 23816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:51.805490 23816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 680fcbc84bba4e7a95eff244c429aa35, State: Initialized, Role: FOLLOWER
I20260929 16:30:51.805881 23816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "680fcbc84bba4e7a95eff244c429aa35" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 32975 } }
I20260929 16:30:51.805991 23816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:51.806074 23816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:51.806187 23816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:51.807080 23816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "680fcbc84bba4e7a95eff244c429aa35" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 32975 } }
I20260929 16:30:51.807395 23816 leader_election.cc:302] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 680fcbc84bba4e7a95eff244c429aa35; no voters:
I20260929 16:30:51.807688 23816 leader_election.cc:288] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:51.807744 23821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:51.807926 23821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [term 1 LEADER]: Becoming Leader. State: Replica: 680fcbc84bba4e7a95eff244c429aa35, State: Running, Role: LEADER
I20260929 16:30:51.808257 23821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "680fcbc84bba4e7a95eff244c429aa35" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 32975 } }
I20260929 16:30:51.808867 23816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:51.809813 23823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 680fcbc84bba4e7a95eff244c429aa35. Latest consensus state: current_term: 1 leader_uuid: "680fcbc84bba4e7a95eff244c429aa35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "680fcbc84bba4e7a95eff244c429aa35" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 32975 } } }
I20260929 16:30:51.810112 23822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "680fcbc84bba4e7a95eff244c429aa35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "680fcbc84bba4e7a95eff244c429aa35" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 32975 } } }
I20260929 16:30:51.810220 23823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:51.810297 23822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:51.810899 23831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:51.813817 23831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:51.819011 23831 catalog_manager.cc:1347] Generated new cluster ID: 41d3c9aaf79c4bd1b5b4f392962b9512
I20260929 16:30:51.819113 23831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 16:30:51 dist-test-slave-mlmz krb5kdc[23735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:51.826534 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:51.803426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.23.44.62:32975)
0929 16:30:51.803864 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:51.803885 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:51.805056 (+ 1171us) server_negotiation.cc:407] Connection header received
0929 16:30:51.808177 (+ 3121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:51.808197 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:51.808530 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:51.808780 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:51.812447 (+ 3667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:51.813941 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:51.815577 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:51.816217 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:51.816366 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:51.821063 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:51.821069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:51.821084 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:51.821111 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:51.823102 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:51.823695 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:51.823700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:51.823705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:51.823826 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:51.824203 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:51.824206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:51.824208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:51.824649 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:51.824835 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:51.825482 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 16:30:51.825744 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":132,"threads_started":1}
I20260929 16:30:51.829571 23831 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:51.830314 23831 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:51.836714 23831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 680fcbc84bba4e7a95eff244c429aa35: Generated new TSK 0
I20260929 16:30:51.837339 23831 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:30:51.866598 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:32975
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:51.997020 23842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:51.997311 23842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:51.997371 23842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:52.001214 23842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:52.001276 23842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:52.001334 23842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:30:52.006331 23842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:32975
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:52.007205 23842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:52.008104 23842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:30:52.017602 23842 server_base.cc:1034] running on GCE node
W20260929 16:30:52.017570 23848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:52.017740 23847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:52.017589 23850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:52.018153 23842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:52.018774 23842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:52.019934 23842 hybrid_clock.cc:648] HybridClock initialized: now 1790699452019922 us; error 25 us; skew 500 ppm
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[23735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699452, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:52.022575 23842 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:30:52.023355 23842 webserver.cc:460] Webserver started at http://127.23.44.1:39783/ using document root <none> and password file <none>
I20260929 16:30:52.023872 23842 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:52.023958 23842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:52.024173 23842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:52.026117 23842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "4a8e8631405d4bf5bb4424db92fe6f9c"
format_stamp: "Formatted at 2026-09-29 16:30:52 on dist-test-slave-mlmz"
server_key: "17a71cca49e4f4f4eeffc761740836c3"
server_key_iv: "75e8ddf055103871762274e50243b270"
server_key_version: "encryptionkey@0"
I20260929 16:30:52.026662 23842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "4a8e8631405d4bf5bb4424db92fe6f9c"
format_stamp: "Formatted at 2026-09-29 16:30:52 on dist-test-slave-mlmz"
server_key: "17a71cca49e4f4f4eeffc761740836c3"
server_key_iv: "75e8ddf055103871762274e50243b270"
server_key_version: "encryptionkey@0"
I20260929 16:30:52.030112 23842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:52.032550 23857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:52.033715 23842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:30:52.033826 23842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "4a8e8631405d4bf5bb4424db92fe6f9c"
format_stamp: "Formatted at 2026-09-29 16:30:52 on dist-test-slave-mlmz"
server_key: "17a71cca49e4f4f4eeffc761740836c3"
server_key_iv: "75e8ddf055103871762274e50243b270"
server_key_version: "encryptionkey@0"
I20260929 16:30:52.033928 23842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:52.047145 23842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:52.049645 23842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:52.049818 23842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:52.050323 23842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:52.051033 23842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:52.051077 23842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:52.051112 23842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:52.051127 23842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:52.059624 23842 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:41269
I20260929 16:30:52.059659 23970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:41269 every 8 connection(s)
I20260929 16:30:52.060418 23842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:52.063077 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 23842
I20260929 16:30:52.063189 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:30:52.063472 23728 external_mini_cluster.cc:1428] Setting key 3d8d36e063cededec4d5ed4b5e221ce9
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[23735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699452, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
I20260929 16:30:52.075721 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:52.062219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:49827 (local address 127.23.44.62:32975)
0929 16:30:52.062354 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:52.062357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:52.063414 (+ 1057us) server_negotiation.cc:407] Connection header received
0929 16:30:52.064368 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:52.064371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:52.064417 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:52.064500 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:52.066001 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.066722 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.067909 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.068369 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.068486 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:52.071680 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:52.071684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:52.071686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:52.071711 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:52.073587 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:52.074119 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:52.074122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:52.074123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:52.074234 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:52.074583 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:52.074585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:52.074586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:52.074760 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:52.074835 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:52.075414 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 16:30:52.075529 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
I20260929 16:30:52.076563 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:52.062487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:32975 (local address 127.23.44.1:49827)
0929 16:30:52.063250 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:52.063304 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:52.064191 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:52.064658 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:52.064664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:52.065079 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:52.065834 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:52.065857 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.066881 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.066884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:52.067772 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:52.067780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.068523 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.068526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:52.068727 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:52.069579 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:52.069595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:52.071497 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:52.073745 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:52.073750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:52.073763 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:52.074020 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:52.074351 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:52.074353 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:52.074355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:52.074488 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:52.074880 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:52.074887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:52.075154 (+ 267us) client_negotiation.cc:769] Sending connection context
0929 16:30:52.075354 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 16:30:52.075637 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":126,"threads_started":1}
I20260929 16:30:52.077903 23971 heartbeater.cc:347] Connected to a master server at 127.23.44.62:32975
I20260929 16:30:52.078159 23971 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:52.078655 23971 heartbeater.cc:511] Master 127.23.44.62:32975 requested a full tablet report, sending...
I20260929 16:30:52.080148 23780 ts_manager.cc:194] Registered new tserver with Master: 4a8e8631405d4bf5bb4424db92fe6f9c (127.23.44.1:41269)
I20260929 16:30:52.081399 23780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:49827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:30:52.091933 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:32975
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:52.204006 23976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:52.204245 23976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:52.204288 23976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:52.208775 23976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:52.208824 23976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:52.208912 23976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:30:52.214147 23976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:32975
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:52.215056 23976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:52.216434 23976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:52.225519 23981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:52.225523 23984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:52.225543 23982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:52.225800 23976 server_base.cc:1034] running on GCE node
I20260929 16:30:52.226269 23976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:52.226845 23976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:52.228031 23976 hybrid_clock.cc:648] HybridClock initialized: now 1790699452228022 us; error 30 us; skew 500 ppm
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[23735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699452, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:52.230698 23976 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:30:52.231483 23976 webserver.cc:460] Webserver started at http://127.23.44.2:35707/ using document root <none> and password file <none>
I20260929 16:30:52.232023 23976 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:52.232111 23976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:52.232360 23976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:52.234366 23976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "16d2428579cc4093a21f2c67f379e50f"
format_stamp: "Formatted at 2026-09-29 16:30:52 on dist-test-slave-mlmz"
server_key: "72c2a538f8b4f06727ee05abdba634b3"
server_key_iv: "f3b1b9bbd2f57523b100b95891bb982d"
server_key_version: "encryptionkey@0"
I20260929 16:30:52.234913 23976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "16d2428579cc4093a21f2c67f379e50f"
format_stamp: "Formatted at 2026-09-29 16:30:52 on dist-test-slave-mlmz"
server_key: "72c2a538f8b4f06727ee05abdba634b3"
server_key_iv: "f3b1b9bbd2f57523b100b95891bb982d"
server_key_version: "encryptionkey@0"
I20260929 16:30:52.238576 23976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:30:52.240906 23991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:52.242316 23976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:30:52.242411 23976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "16d2428579cc4093a21f2c67f379e50f"
format_stamp: "Formatted at 2026-09-29 16:30:52 on dist-test-slave-mlmz"
server_key: "72c2a538f8b4f06727ee05abdba634b3"
server_key_iv: "f3b1b9bbd2f57523b100b95891bb982d"
server_key_version: "encryptionkey@0"
I20260929 16:30:52.242493 23976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:52.254734 23976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:52.257956 23976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:52.258100 23976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:52.258692 23976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:52.259431 23976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:52.259476 23976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:52.259516 23976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:52.259536 23976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:52.267995 23976 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:46227
I20260929 16:30:52.268019 24104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:46227 every 8 connection(s)
I20260929 16:30:52.268867 23976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:30:52.277844 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 23976
I20260929 16:30:52.277967 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[23735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699452, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:52.278283 23728 external_mini_cluster.cc:1428] Setting key 58e88f12d29eda4d0dc42f81f18c1e99
I20260929 16:30:52.282470 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:52.270596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:35441 (local address 127.23.44.62:32975)
0929 16:30:52.270719 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:52.270723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:52.271826 (+ 1103us) server_negotiation.cc:407] Connection header received
0929 16:30:52.272818 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:52.272821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:52.272876 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:52.272984 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:52.274152 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.274691 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.275810 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.276032 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.276143 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:52.278787 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:52.278791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:52.278794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:52.278819 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:52.280264 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:52.280871 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:52.280874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:52.280875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:52.280944 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:52.281322 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:52.281325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:52.281326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:52.281538 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:52.281658 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:52.282150 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 16:30:52.282285 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:30:52.283142 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:52.270966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:32975 (local address 127.23.44.2:35441)
0929 16:30:52.271687 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:52.271726 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:52.272622 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:52.273154 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:52.273161 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:52.273555 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:52.274007 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:52.274027 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.274815 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.274823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:52.275689 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:52.275697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.276176 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.276180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:52.276345 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:52.276881 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:52.276896 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:52.278607 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:52.280378 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:52.280387 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:52.280401 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:52.280770 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:52.281030 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:52.281036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:52.281041 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:52.281220 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:52.281661 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:52.281671 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:52.281915 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 16:30:52.282059 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 16:30:52.282282 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":122,"threads_started":1}
I20260929 16:30:52.284207 24105 heartbeater.cc:347] Connected to a master server at 127.23.44.62:32975
I20260929 16:30:52.284443 24105 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:52.285043 24105 heartbeater.cc:511] Master 127.23.44.62:32975 requested a full tablet report, sending...
I20260929 16:30:52.286934 23780 ts_manager.cc:194] Registered new tserver with Master: 16d2428579cc4093a21f2c67f379e50f (127.23.44.2:46227)
I20260929 16:30:52.287743 23780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:35441
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:30:52.306123 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:32975
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:52.413070 24110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:52.413307 24110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:52.413348 24110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:52.417331 24110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:52.417380 24110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:52.417436 24110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:30:52.422585 24110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:32975
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:52.423391 24110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:52.424232 24110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:52.433601 24118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:52.433641 24116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:52.433804 24110 server_base.cc:1034] running on GCE node
W20260929 16:30:52.433599 24115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:52.434324 24110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:52.434940 24110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:52.436124 24110 hybrid_clock.cc:648] HybridClock initialized: now 1790699452436113 us; error 24 us; skew 500 ppm
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[23735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699452, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:52.438868 24110 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:30:52.439654 24110 webserver.cc:460] Webserver started at http://127.23.44.3:35157/ using document root <none> and password file <none>
I20260929 16:30:52.440171 24110 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:52.440258 24110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:52.440471 24110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:52.442386 24110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "ed5c0a9fb1694443b9570d463cc9327e"
format_stamp: "Formatted at 2026-09-29 16:30:52 on dist-test-slave-mlmz"
server_key: "97c238bd1b171833320e6b9b2923c51e"
server_key_iv: "4a055302c14d2c59b593ae2d917893b1"
server_key_version: "encryptionkey@0"
I20260929 16:30:52.442925 24110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "ed5c0a9fb1694443b9570d463cc9327e"
format_stamp: "Formatted at 2026-09-29 16:30:52 on dist-test-slave-mlmz"
server_key: "97c238bd1b171833320e6b9b2923c51e"
server_key_iv: "4a055302c14d2c59b593ae2d917893b1"
server_key_version: "encryptionkey@0"
I20260929 16:30:52.446578 24110 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:30:52.448974 24125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:52.450156 24110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:30:52.450292 24110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "ed5c0a9fb1694443b9570d463cc9327e"
format_stamp: "Formatted at 2026-09-29 16:30:52 on dist-test-slave-mlmz"
server_key: "97c238bd1b171833320e6b9b2923c51e"
server_key_iv: "4a055302c14d2c59b593ae2d917893b1"
server_key_version: "encryptionkey@0"
I20260929 16:30:52.450397 24110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:52.460712 24110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:52.463897 24110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:52.464083 24110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:52.464607 24110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:52.465409 24110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:52.465473 24110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:52.465534 24110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:52.465584 24110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:52.474170 24110 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:41689
I20260929 16:30:52.474198 24238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:41689 every 8 connection(s)
I20260929 16:30:52.474973 24110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:30:52.481449 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 24110
I20260929 16:30:52.481582 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:30:52.481819 23728 external_mini_cluster.cc:1428] Setting key bde81297313d3219182441b10309ef34
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[23735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699452, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:52.487573 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:52.476706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:33829 (local address 127.23.44.62:32975)
0929 16:30:52.476843 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:52.476847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:52.477836 (+ 989us) server_negotiation.cc:407] Connection header received
0929 16:30:52.478626 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:52.478628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:52.478671 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:52.478747 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:52.479829 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.480352 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.481592 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.481902 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.482003 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:52.484575 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:52.484579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:52.484580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:52.484606 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:52.485933 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:52.486378 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:52.486380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:52.486382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:52.486460 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:52.486732 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:52.486734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:52.486735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:52.486891 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:52.486962 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:52.487375 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 16:30:52.487462 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:30:52.488297 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:52.477037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:32975 (local address 127.23.44.3:33829)
0929 16:30:52.477691 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:52.477730 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:52.478447 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:52.478891 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:52.478897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:52.479243 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:52.479693 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:52.479706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.480474 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.480476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:52.481439 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:52.481448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.482036 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.482040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:52.482237 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:52.482870 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:52.482886 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:52.484423 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:52.486031 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:52.486035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:52.486046 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:52.486291 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:52.486536 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:52.486538 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:52.486540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:52.486658 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:52.486974 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:52.486978 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:52.487184 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 16:30:52.487329 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 16:30:52.487545 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":115,"threads_started":1}
I20260929 16:30:52.489562 24239 heartbeater.cc:347] Connected to a master server at 127.23.44.62:32975
I20260929 16:30:52.489795 24239 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:52.490227 24239 heartbeater.cc:511] Master 127.23.44.62:32975 requested a full tablet report, sending...
I20260929 16:30:52.491181 23780 ts_manager.cc:194] Registered new tserver with Master: ed5c0a9fb1694443b9570d463cc9327e (127.23.44.3:41689)
I20260929 16:30:52.491706 23780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:33829
I20260929 16:30:52.495747 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[23735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699452, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[23735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699452, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:30:52.520689 24246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:52.511539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.23.44.1:41269)
0929 16:30:52.511850 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:52.511859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:52.511882 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:30:52.512034 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:52.512040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:52.512248 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:52.512367 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:52.512954 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.513851 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.514688 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.515030 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.515137 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:52.517492 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:52.517500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:52.517512 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:52.517533 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:52.519018 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:52.519431 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:52.519436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:52.519437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:52.519513 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:52.519775 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:52.519777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:52.519778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:52.520000 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:52.520169 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:52.520351 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 16:30:52.520503 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":133,"threads_started":1}
W20260929 16:30:52.521628 23885 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60454
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[23735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[23735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:30:52.546205 24246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:52.538176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.23.44.1:41269)
0929 16:30:52.538354 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:52.538359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:52.538480 (+ 121us) server_negotiation.cc:407] Connection header received
0929 16:30:52.538634 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:52.538637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:52.538684 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:52.538767 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:52.539422 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.539915 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.540715 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:52.540944 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:52.541057 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:52.543299 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:52.543302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:52.543304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:52.543326 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:52.544553 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:52.544982 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:52.544985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:52.544986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:52.545064 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:52.545420 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:52.545423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:52.545425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:52.545598 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:52.545719 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:52.545957 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 16:30:52.546057 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 16:30:52.547286 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 23842
I20260929 16:30:52.554919 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 23976
I20260929 16:30:52.560979 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 24110
I20260929 16:30:52.566336 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 23748
2026-09-29T16:30:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2135 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[24253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[24253](info): set up 2 sockets
Sep 29 16:30:52 dist-test-slave-mlmz krb5kdc[24253](info): commencing operation
krb5kdc: starting...
W20260929 16:30:53.605378 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:30:53 dist-test-slave-mlmz krb5kdc[24253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:30:53.726531 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:43237
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:43237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:53.831496 24266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:53.831725 24266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:53.831763 24266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:53.835651 24266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:53.835696 24266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:53.835716 24266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:53.835731 24266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:53.835747 24266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:53.840688 24266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:43237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:43237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:53.841593 24266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:53.842519 24266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:53.850428 24272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:53.850422 24266 server_base.cc:1034] running on GCE node
W20260929 16:30:53.850432 24271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:53.850699 24274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:53.851176 24266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:53.852131 24266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:53.853317 24266 hybrid_clock.cc:648] HybridClock initialized: now 1790699453853300 us; error 28 us; skew 500 ppm
Sep 29 16:30:53 dist-test-slave-mlmz krb5kdc[24253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699453, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:53.855837 24266 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:30:53.856653 24266 webserver.cc:460] Webserver started at http://127.23.44.62:40587/ using document root <none> and password file <none>
I20260929 16:30:53.857167 24266 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:53.857254 24266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:53.857494 24266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:53.859335 24266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "9b137ad5566b4707bdaf69f637783e34"
format_stamp: "Formatted at 2026-09-29 16:30:53 on dist-test-slave-mlmz"
server_key: "5d042bdc70a86da49b29156e0a6c6720"
server_key_iv: "642ee69733bacaf787197c51a6e85301"
server_key_version: "encryptionkey@0"
I20260929 16:30:53.859865 24266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "9b137ad5566b4707bdaf69f637783e34"
format_stamp: "Formatted at 2026-09-29 16:30:53 on dist-test-slave-mlmz"
server_key: "5d042bdc70a86da49b29156e0a6c6720"
server_key_iv: "642ee69733bacaf787197c51a6e85301"
server_key_version: "encryptionkey@0"
I20260929 16:30:53.863453 24266 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:30:53.865775 24281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:53.866899 24266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:30:53.867041 24266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "9b137ad5566b4707bdaf69f637783e34"
format_stamp: "Formatted at 2026-09-29 16:30:53 on dist-test-slave-mlmz"
server_key: "5d042bdc70a86da49b29156e0a6c6720"
server_key_iv: "642ee69733bacaf787197c51a6e85301"
server_key_version: "encryptionkey@0"
I20260929 16:30:53.867149 24266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:53.882369 24266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:53.884999 24266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:53.885195 24266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:53.891654 24266 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:43237
I20260929 16:30:53.891665 24333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:43237 every 8 connection(s)
I20260929 16:30:53.892508 24266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:53.895022 24334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:53.899955 24334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34: Bootstrap starting.
I20260929 16:30:53.902177 24334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:53.902285 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 24266
I20260929 16:30:53.902402 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:30:53.902661 23728 external_mini_cluster.cc:1428] Setting key 772e01f65a82478eb1033f4420464d0a
I20260929 16:30:53.903039 24334 log.cc:824] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:53.904842 24334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34: No bootstrap required, opened a new log
I20260929 16:30:53.908861 24334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b137ad5566b4707bdaf69f637783e34" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43237 } }
I20260929 16:30:53.909144 24334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:53.909209 24334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b137ad5566b4707bdaf69f637783e34, State: Initialized, Role: FOLLOWER
I20260929 16:30:53.909750 24334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b137ad5566b4707bdaf69f637783e34" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43237 } }
I20260929 16:30:53.909909 24334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:53.909977 24334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:53.910086 24334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:53.911094 24334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b137ad5566b4707bdaf69f637783e34" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43237 } }
Sep 29 16:30:53 dist-test-slave-mlmz krb5kdc[24253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:53.911406 24334 leader_election.cc:302] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b137ad5566b4707bdaf69f637783e34; no voters:
I20260929 16:30:53.911715 24334 leader_election.cc:288] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:53.911810 24339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:53.912034 24339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [term 1 LEADER]: Becoming Leader. State: Replica: 9b137ad5566b4707bdaf69f637783e34, State: Running, Role: LEADER
I20260929 16:30:53.912405 24339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b137ad5566b4707bdaf69f637783e34" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43237 } }
I20260929 16:30:53.912957 24334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:53.914321 24341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b137ad5566b4707bdaf69f637783e34. Latest consensus state: current_term: 1 leader_uuid: "9b137ad5566b4707bdaf69f637783e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b137ad5566b4707bdaf69f637783e34" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43237 } } }
I20260929 16:30:53.914330 24340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b137ad5566b4707bdaf69f637783e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b137ad5566b4707bdaf69f637783e34" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43237 } } }
I20260929 16:30:53.914554 24341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:53.914552 24340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:53.917714 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:53.903714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.23.44.62:43237)
0929 16:30:53.904103 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:53.904123 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:53.904161 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:30:53.904748 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:53.904763 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:53.905101 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:53.905415 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:53.906194 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:53.907141 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:53.908396 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:53.908839 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:53.908950 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:53.912165 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:53.912171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:53.912187 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:53.912216 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:53.914480 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:53.915006 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:53.915012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:53.915018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:53.915179 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:53.915724 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:53.915729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:53.915731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:53.916099 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:53.916306 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:53.916600 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 16:30:53.916904 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
W20260929 16:30:53.918167 24353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:30:53.918242 24353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:30:53.918315 24354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:53.919113 24354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:53.924275 24354 catalog_manager.cc:1347] Generated new cluster ID: 58cc8e895d0b4d2094427c6cd0a3005b
I20260929 16:30:53.924347 24354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:53.931423 24354 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:53.932287 24354 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:53.938191 24354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b137ad5566b4707bdaf69f637783e34: Generated new TSK 0
I20260929 16:30:53.938947 24354 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:30:53.972640 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:43237
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:54.078438 24360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:54.078685 24360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:54.078753 24360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:54.082659 24360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:54.082732 24360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:54.082815 24360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:30:54.087939 24360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:43237
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:54.088958 24360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:54.089937 24360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:54.099066 24365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:54.099090 24368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:54.099231 24360 server_base.cc:1034] running on GCE node
W20260929 16:30:54.099067 24366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:54.099782 24360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:54.100378 24360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:54.101534 24360 hybrid_clock.cc:648] HybridClock initialized: now 1790699454101519 us; error 25 us; skew 500 ppm
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699454, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:54.104386 24360 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:30:54.105197 24360 webserver.cc:460] Webserver started at http://127.23.44.1:43813/ using document root <none> and password file <none>
I20260929 16:30:54.105767 24360 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:54.105827 24360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:54.105991 24360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:54.107909 24360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "b38c8e537f8546699e6cdc4750f7603d"
format_stamp: "Formatted at 2026-09-29 16:30:54 on dist-test-slave-mlmz"
server_key: "8d56bf3ca2778dbe695e148599552a2e"
server_key_iv: "021f6ccd3d1f58c8a95796cf0590143d"
server_key_version: "encryptionkey@0"
I20260929 16:30:54.108402 24360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "b38c8e537f8546699e6cdc4750f7603d"
format_stamp: "Formatted at 2026-09-29 16:30:54 on dist-test-slave-mlmz"
server_key: "8d56bf3ca2778dbe695e148599552a2e"
server_key_iv: "021f6ccd3d1f58c8a95796cf0590143d"
server_key_version: "encryptionkey@0"
I20260929 16:30:54.111791 24360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 16:30:54.114148 24375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:54.115388 24360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 16:30:54.115502 24360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "b38c8e537f8546699e6cdc4750f7603d"
format_stamp: "Formatted at 2026-09-29 16:30:54 on dist-test-slave-mlmz"
server_key: "8d56bf3ca2778dbe695e148599552a2e"
server_key_iv: "021f6ccd3d1f58c8a95796cf0590143d"
server_key_version: "encryptionkey@0"
I20260929 16:30:54.115599 24360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:54.128762 24360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:54.131356 24360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:54.131539 24360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:54.132058 24360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:54.132800 24360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:54.132844 24360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:54.132910 24360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:54.132942 24360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:54.141381 24360 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:44619
I20260929 16:30:54.141462 24488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:44619 every 8 connection(s)
I20260929 16:30:54.142225 24360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:54.148273 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 24360
I20260929 16:30:54.148379 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:30:54.148617 23728 external_mini_cluster.cc:1428] Setting key a77c9516885da79443743eafb37f0004
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699454, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:54.156147 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.143900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:52781 (local address 127.23.44.62:43237)
0929 16:30:54.144043 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:54.144048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:54.144868 (+ 820us) server_negotiation.cc:407] Connection header received
0929 16:30:54.145754 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:54.145756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:54.145800 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:54.145871 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:54.146950 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.147510 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.148976 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.149349 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.149429 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.152123 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:54.152130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:54.152135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:54.152166 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:54.153856 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.154454 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.154460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.154465 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.154562 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.154936 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.154942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.154947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.155210 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:54.155320 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:54.155757 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 16:30:54.155909 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:30:54.156667 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.144290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:43237 (local address 127.23.44.1:52781)
0929 16:30:54.144714 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:54.144753 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:54.145566 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:54.146023 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:54.146029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:54.146377 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:54.146806 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.146819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.147639 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.147642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.148847 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.148858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.149494 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.149498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.149678 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.150294 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:54.150309 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:54.151970 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:54.153984 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:54.154008 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:54.154023 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:54.154342 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:54.154664 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:54.154667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:54.154669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:54.154834 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:54.155331 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:54.155335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:54.155533 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 16:30:54.155672 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 16:30:54.155898 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
I20260929 16:30:54.157842 24489 heartbeater.cc:347] Connected to a master server at 127.23.44.62:43237
I20260929 16:30:54.158057 24489 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
I20260929 16:30:54.158521 24489 heartbeater.cc:511] Master 127.23.44.62:43237 requested a full tablet report, sending...
I20260929 16:30:54.159854 24298 ts_manager.cc:194] Registered new tserver with Master: b38c8e537f8546699e6cdc4750f7603d (127.23.44.1:44619)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
I20260929 16:30:54.161013 24298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:52781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:30:54.175920 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:43237
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:54.280867 24494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:54.281129 24494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:54.281206 24494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:54.285053 24494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:54.285130 24494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:54.285228 24494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:30:54.290572 24494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:43237
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:54.291498 24494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:54.292411 24494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:54.302454 24502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:54.302623 24499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:54.302721 24494 server_base.cc:1034] running on GCE node
W20260929 16:30:54.302624 24500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:54.303215 24494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:54.303797 24494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:54.304980 24494 hybrid_clock.cc:648] HybridClock initialized: now 1790699454304946 us; error 46 us; skew 500 ppm
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699454, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:54.307582 24494 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:30:54.308472 24494 webserver.cc:460] Webserver started at http://127.23.44.2:36359/ using document root <none> and password file <none>
I20260929 16:30:54.309005 24494 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:54.309092 24494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:54.309360 24494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:54.311321 24494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "49285128f79f43a2a889e7b5d16c1d72"
format_stamp: "Formatted at 2026-09-29 16:30:54 on dist-test-slave-mlmz"
server_key: "af3d8804f0392fb0cedf7778a8a22e6f"
server_key_iv: "724193afb03387cf345d719b5d4f6df7"
server_key_version: "encryptionkey@0"
I20260929 16:30:54.311854 24494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "49285128f79f43a2a889e7b5d16c1d72"
format_stamp: "Formatted at 2026-09-29 16:30:54 on dist-test-slave-mlmz"
server_key: "af3d8804f0392fb0cedf7778a8a22e6f"
server_key_iv: "724193afb03387cf345d719b5d4f6df7"
server_key_version: "encryptionkey@0"
I20260929 16:30:54.315456 24494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:30:54.317754 24509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:54.318935 24494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:30:54.319062 24494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "49285128f79f43a2a889e7b5d16c1d72"
format_stamp: "Formatted at 2026-09-29 16:30:54 on dist-test-slave-mlmz"
server_key: "af3d8804f0392fb0cedf7778a8a22e6f"
server_key_iv: "724193afb03387cf345d719b5d4f6df7"
server_key_version: "encryptionkey@0"
I20260929 16:30:54.319197 24494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:54.339200 24494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:54.341923 24494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:54.342109 24494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:54.342638 24494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:54.343405 24494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:54.343472 24494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:54.343540 24494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:54.343585 24494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:54.352466 24494 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:32887
I20260929 16:30:54.352524 24622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:32887 every 8 connection(s)
I20260929 16:30:54.353367 24494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:30:54.361325 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 24494
I20260929 16:30:54.361454 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:30:54.361822 23728 external_mini_cluster.cc:1428] Setting key 8517a22eda13059ae4f55d5282880445
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699454, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:54.366837 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.355152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:50489 (local address 127.23.44.62:43237)
0929 16:30:54.355323 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:54.355329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:54.356207 (+ 878us) server_negotiation.cc:407] Connection header received
0929 16:30:54.356992 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:54.356994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:54.357042 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:54.357121 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:54.358212 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.358800 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.359941 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.360163 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.360259 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.362920 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:54.362924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:54.362926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:54.362948 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:54.364405 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.365045 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.365052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.365055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.365187 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.365609 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.365614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.365616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.365861 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:54.365977 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:54.366461 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 16:30:54.366597 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:30:54.367465 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.355597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:43237 (local address 127.23.44.2:50489)
0929 16:30:54.356046 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:54.356084 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:54.356802 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:54.357294 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:54.357300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:54.357635 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:54.358068 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.358081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.358950 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.358954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.359813 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.359823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.360296 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.360299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.360431 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.360956 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:54.360972 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:54.362759 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:54.364522 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:54.364532 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:54.364549 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:54.364923 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:54.365335 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:54.365338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:54.365340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:54.365513 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:54.365983 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:54.365988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:54.366198 (+ 210us) client_negotiation.cc:769] Sending connection context
0929 16:30:54.366358 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 16:30:54.366591 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":116,"threads_started":1}
I20260929 16:30:54.368525 24623 heartbeater.cc:347] Connected to a master server at 127.23.44.62:43237
I20260929 16:30:54.368754 24623 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:54.369194 24623 heartbeater.cc:511] Master 127.23.44.62:43237 requested a full tablet report, sending...
I20260929 16:30:54.370180 24298 ts_manager.cc:194] Registered new tserver with Master: 49285128f79f43a2a889e7b5d16c1d72 (127.23.44.2:32887)
I20260929 16:30:54.370760 24298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:50489
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:30:54.388154 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:43237
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:54.494665 24628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:54.494910 24628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:54.494952 24628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:54.499200 24628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:54.499251 24628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:54.499308 24628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:30:54.504323 24628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:43237
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:54.505184 24628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:54.506129 24628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:54.515244 24633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:54.515283 24636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:54.515230 24634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:54.515460 24628 server_base.cc:1034] running on GCE node
I20260929 16:30:54.515885 24628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:54.516500 24628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:54.517680 24628 hybrid_clock.cc:648] HybridClock initialized: now 1790699454517667 us; error 25 us; skew 500 ppm
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699454, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:54.520370 24628 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:30:54.521183 24628 webserver.cc:460] Webserver started at http://127.23.44.3:35579/ using document root <none> and password file <none>
I20260929 16:30:54.521765 24628 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:54.521857 24628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:54.522079 24628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:54.524080 24628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "c9bc9536c47c4335bac7dfd0cc9651bb"
format_stamp: "Formatted at 2026-09-29 16:30:54 on dist-test-slave-mlmz"
server_key: "b3258ba11d191ff77efb9f86a1178a80"
server_key_iv: "48c04c410fde0d62bfcad0694466fab4"
server_key_version: "encryptionkey@0"
I20260929 16:30:54.524631 24628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "c9bc9536c47c4335bac7dfd0cc9651bb"
format_stamp: "Formatted at 2026-09-29 16:30:54 on dist-test-slave-mlmz"
server_key: "b3258ba11d191ff77efb9f86a1178a80"
server_key_iv: "48c04c410fde0d62bfcad0694466fab4"
server_key_version: "encryptionkey@0"
I20260929 16:30:54.528090 24628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:30:54.530352 24643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:54.531493 24628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:30:54.531633 24628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "c9bc9536c47c4335bac7dfd0cc9651bb"
format_stamp: "Formatted at 2026-09-29 16:30:54 on dist-test-slave-mlmz"
server_key: "b3258ba11d191ff77efb9f86a1178a80"
server_key_iv: "48c04c410fde0d62bfcad0694466fab4"
server_key_version: "encryptionkey@0"
I20260929 16:30:54.531744 24628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:54.542281 24628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:54.544847 24628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:54.545029 24628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:54.545594 24628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:54.546322 24628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:54.546399 24628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:54.546465 24628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:54.546500 24628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:54.556699 24628 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:42937
I20260929 16:30:54.556715 24756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:42937 every 8 connection(s)
I20260929 16:30:54.557511 24628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:30:54.563349 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 24628
I20260929 16:30:54.563478 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:30:54.563717 23728 external_mini_cluster.cc:1428] Setting key 990fa18b373335dd54d1b5ac8b3da0aa
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699454, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:54.569960 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.559180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:58779 (local address 127.23.44.62:43237)
0929 16:30:54.559329 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:54.559334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:54.560139 (+ 805us) server_negotiation.cc:407] Connection header received
0929 16:30:54.560928 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:54.560930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:54.560974 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:54.561045 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:54.562140 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.562663 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.563911 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.564217 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.564317 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.566876 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:54.566879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:54.566881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:54.566902 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:54.568307 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.568724 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.568727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.568728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.568799 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.569064 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.569066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.569067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.569220 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:54.569373 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:54.569765 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 16:30:54.569851 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:30:54.570755 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.559570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:43237 (local address 127.23.44.3:58779)
0929 16:30:54.559989 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:54.560027 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:54.560747 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:54.561189 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:54.561195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:54.561585 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:54.562018 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.562030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.562766 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.562768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.563776 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.563802 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.564388 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.564392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.564609 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.565224 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:54.565252 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:54.566741 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:54.568404 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:54.568409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:54.568418 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:54.568639 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:54.568874 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:54.568877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:54.568878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:54.568991 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:54.569379 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:54.569384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:54.569574 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 16:30:54.569715 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 16:30:54.569929 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260929 16:30:54.571843 24757 heartbeater.cc:347] Connected to a master server at 127.23.44.62:43237
I20260929 16:30:54.572052 24757 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:54.572482 24757 heartbeater.cc:511] Master 127.23.44.62:43237 requested a full tablet report, sending...
I20260929 16:30:54.573468 24298 ts_manager.cc:194] Registered new tserver with Master: c9bc9536c47c4335bac7dfd0cc9651bb (127.23.44.3:42937)
I20260929 16:30:54.573998 24298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:58779
I20260929 16:30:54.577178 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:30:54.586999 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.580269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.23.44.62:43237)
0929 16:30:54.580399 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:54.580402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:54.580460 (+ 58us) server_negotiation.cc:407] Connection header received
0929 16:30:54.580549 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:54.580551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:54.580582 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:54.580644 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:54.581319 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.581787 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.582601 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.582794 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.582864 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.584062 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:54.584065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:54.584067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:54.584086 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:54.585454 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.585861 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.585863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.585864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.585932 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.586236 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.586239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.586240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.586427 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:54.586528 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:54.586720 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 16:30:54.586808 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:30:54.591118 24298 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:30:54.593880 24298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:30:54.607702 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.601954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:32887 (local address 127.0.0.1:57604)
0929 16:30:54.602857 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:54.602869 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:54.602950 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:54.603441 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:54.603444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:54.603462 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:54.603550 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.603557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.604579 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.604586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.606075 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.606086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.607165 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.607169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.607274 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.607310 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:30:54.607415 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 16:30:54.607532 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":69,"thread_start_us":87,"threads_started":1}
I20260929 16:30:54.607894 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.601689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:42937 (local address 127.0.0.1:34864)
0929 16:30:54.602265 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:54.602311 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:54.602434 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:54.602981 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:54.602988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:54.603009 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:54.603156 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.603167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.604809 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.604812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.606075 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.606086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.607489 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.607492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.607583 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.607604 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:30:54.607688 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 16:30:54.607766 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":24,"thread_start_us":118,"threads_started":1}
I20260929 16:30:54.608098 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.601690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.23.44.1:44619)
0929 16:30:54.602316 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:54.602328 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:54.602547 (+ 219us) server_negotiation.cc:407] Connection header received
0929 16:30:54.602751 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:54.602756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:54.602914 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:54.603026 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:54.603457 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.604674 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.606538 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.607082 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.607243 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.607378 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:54.607650 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 16:30:54.607861 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":119,"threads_started":1}
I20260929 16:30:54.608161 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.602212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.23.44.2:32887)
0929 16:30:54.602731 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:54.602743 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:54.603045 (+ 302us) server_negotiation.cc:407] Connection header received
0929 16:30:54.603121 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:54.603128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:54.603305 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:54.603451 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:54.603676 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.604462 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.606201 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.607036 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.607146 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.607287 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:54.607700 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 16:30:54.607939 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":132,"threads_started":1}
I20260929 16:30:54.608326 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.602172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.23.44.3:42937)
0929 16:30:54.602494 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:54.602507 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:54.602530 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:30:54.602606 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:54.602611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:54.602806 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:54.603302 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:54.603468 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.604673 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.606577 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.607390 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.607488 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.607620 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:54.608053 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 16:30:54.608150 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
I20260929 16:30:54.608642 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.601493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:44619 (local address 127.0.0.1:41676)
0929 16:30:54.602440 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:54.602454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:54.602616 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:54.603203 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:54.603208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:54.603229 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:54.603316 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.603322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.605345 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.605350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.606415 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.606426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.607368 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.607372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.607475 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.607498 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:30:54.608292 (+ 794us) client_negotiation.cc:240] Negotiation successful
0929 16:30:54.608372 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":65,"threads_started":1}
I20260929 16:30:54.610838 24557 tablet_service.cc:1467] Processing CreateTablet for tablet 3195d35b4e9e4508a5d514141d2b0e7e (DEFAULT_TABLE table=test-table [id=8dadd8888c6e4c1e965dd04298c54117]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:54.611002 24423 tablet_service.cc:1467] Processing CreateTablet for tablet 3195d35b4e9e4508a5d514141d2b0e7e (DEFAULT_TABLE table=test-table [id=8dadd8888c6e4c1e965dd04298c54117]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:54.611002 24691 tablet_service.cc:1467] Processing CreateTablet for tablet 3195d35b4e9e4508a5d514141d2b0e7e (DEFAULT_TABLE table=test-table [id=8dadd8888c6e4c1e965dd04298c54117]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:54.611811 24557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3195d35b4e9e4508a5d514141d2b0e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:54.611975 24691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3195d35b4e9e4508a5d514141d2b0e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:54.611975 24423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3195d35b4e9e4508a5d514141d2b0e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:54.618139 24774 tablet_bootstrap.cc:492] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72: Bootstrap starting.
I20260929 16:30:54.619552 24775 tablet_bootstrap.cc:492] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d: Bootstrap starting.
I20260929 16:30:54.619546 24776 tablet_bootstrap.cc:492] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb: Bootstrap starting.
I20260929 16:30:54.620627 24774 tablet_bootstrap.cc:654] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:54.621381 24774 log.cc:824] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:54.621644 24775 tablet_bootstrap.cc:654] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:54.621644 24776 tablet_bootstrap.cc:654] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:54.622884 24775 log.cc:824] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:54.622884 24776 log.cc:824] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:54.623366 24774 tablet_bootstrap.cc:492] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72: No bootstrap required, opened a new log
I20260929 16:30:54.623500 24774 ts_tablet_manager.cc:1397] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:30:54.624548 24776 tablet_bootstrap.cc:492] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb: No bootstrap required, opened a new log
I20260929 16:30:54.624742 24776 ts_tablet_manager.cc:1397] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:30:54.624738 24775 tablet_bootstrap.cc:492] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d: No bootstrap required, opened a new log
I20260929 16:30:54.624918 24775 ts_tablet_manager.cc:1397] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:30:54.627257 24774 raft_consensus.cc:348] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38c8e537f8546699e6cdc4750f7603d" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44619 } } peers { permanent_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42937 } } peers { permanent_uuid: "49285128f79f43a2a889e7b5d16c1d72" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32887 } }
I20260929 16:30:54.627529 24774 raft_consensus.cc:374] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:54.627583 24774 raft_consensus.cc:729] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49285128f79f43a2a889e7b5d16c1d72, State: Initialized, Role: FOLLOWER
I20260929 16:30:54.627451 24776 raft_consensus.cc:348] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38c8e537f8546699e6cdc4750f7603d" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44619 } } peers { permanent_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42937 } } peers { permanent_uuid: "49285128f79f43a2a889e7b5d16c1d72" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32887 } }
I20260929 16:30:54.627751 24776 raft_consensus.cc:374] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:54.627811 24776 raft_consensus.cc:729] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9bc9536c47c4335bac7dfd0cc9651bb, State: Initialized, Role: FOLLOWER
I20260929 16:30:54.628010 24774 consensus_queue.cc:260] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38c8e537f8546699e6cdc4750f7603d" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44619 } } peers { permanent_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42937 } } peers { permanent_uuid: "49285128f79f43a2a889e7b5d16c1d72" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32887 } }
I20260929 16:30:54.628234 24776 consensus_queue.cc:260] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38c8e537f8546699e6cdc4750f7603d" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44619 } } peers { permanent_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42937 } } peers { permanent_uuid: "49285128f79f43a2a889e7b5d16c1d72" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32887 } }
I20260929 16:30:54.628528 24775 raft_consensus.cc:348] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38c8e537f8546699e6cdc4750f7603d" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44619 } } peers { permanent_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42937 } } peers { permanent_uuid: "49285128f79f43a2a889e7b5d16c1d72" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32887 } }
I20260929 16:30:54.628700 24623 heartbeater.cc:503] Master 127.23.44.62:43237 was elected leader, sending a full tablet report...
I20260929 16:30:54.628744 24775 raft_consensus.cc:374] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:54.628798 24775 raft_consensus.cc:729] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38c8e537f8546699e6cdc4750f7603d, State: Initialized, Role: FOLLOWER
I20260929 16:30:54.628989 24774 ts_tablet_manager.cc:1428] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:30:54.629108 24776 ts_tablet_manager.cc:1428] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:54.629433 24757 heartbeater.cc:503] Master 127.23.44.62:43237 was elected leader, sending a full tablet report...
I20260929 16:30:54.629194 24775 consensus_queue.cc:260] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38c8e537f8546699e6cdc4750f7603d" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44619 } } peers { permanent_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42937 } } peers { permanent_uuid: "49285128f79f43a2a889e7b5d16c1d72" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32887 } }
I20260929 16:30:54.631119 24775 ts_tablet_manager.cc:1428] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:30:54.631141 24489 heartbeater.cc:503] Master 127.23.44.62:43237 was elected leader, sending a full tablet report...
W20260929 16:30:54.643280 24490 tablet.cc:2369] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:30:54.808472 24758 tablet.cc:2369] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:30:54.811101 24781 raft_consensus.cc:482] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:54.811246 24781 raft_consensus.cc:504] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38c8e537f8546699e6cdc4750f7603d" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44619 } } peers { permanent_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42937 } } peers { permanent_uuid: "49285128f79f43a2a889e7b5d16c1d72" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32887 } }
I20260929 16:30:54.812314 24781 leader_election.cc:288] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b38c8e537f8546699e6cdc4750f7603d (127.23.44.1:44619), 49285128f79f43a2a889e7b5d16c1d72 (127.23.44.2:32887)
I20260929 16:30:54.817173 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.812765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:56681 (local address 127.23.44.2:32887)
0929 16:30:54.812869 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:54.812875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:54.813045 (+ 170us) server_negotiation.cc:407] Connection header received
0929 16:30:54.813194 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:54.813196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:54.813235 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:54.813346 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:54.813597 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.814331 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.815923 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.816531 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.816639 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.816701 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:54.816979 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 16:30:54.817078 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 16:30:54.817169 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.812614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:36753 (local address 127.23.44.1:44619)
0929 16:30:54.812751 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:54.812756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:54.812846 (+ 90us) server_negotiation.cc:407] Connection header received
0929 16:30:54.812984 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:54.812987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:54.813027 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:54.813107 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:54.813415 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.814235 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.815923 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.816531 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.816639 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.816701 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:54.816979 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 16:30:54.817049 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 16:30:54.817190 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.812530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:44619 (local address 127.23.44.3:36753)
0929 16:30:54.812747 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:54.812763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:54.812861 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:54.813119 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:54.813122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:54.813137 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:54.813298 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.813309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.814348 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.814352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.815795 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.815805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.816648 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.816652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.816827 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.816867 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:30:54.816945 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:30:54.817043 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":138}
I20260929 16:30:54.817199 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.812568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:32887 (local address 127.23.44.3:56681)
0929 16:30:54.812942 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:54.812953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:54.813074 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:54.813366 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:54.813372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:54.813393 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:54.813498 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.813503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.814427 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.814430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.815800 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:54.815805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.816648 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.816652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:54.816831 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.816871 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:30:54.816945 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:30:54.817043 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":122,"thread_start_us":132,"threads_started":1}
I20260929 16:30:54.817935 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3195d35b4e9e4508a5d514141d2b0e7e" candidate_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38c8e537f8546699e6cdc4750f7603d" is_pre_election: true
I20260929 16:30:54.818065 24577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3195d35b4e9e4508a5d514141d2b0e7e" candidate_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49285128f79f43a2a889e7b5d16c1d72" is_pre_election: true
I20260929 16:30:54.818269 24577 raft_consensus.cc:2454] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9bc9536c47c4335bac7dfd0cc9651bb in term 0.
I20260929 16:30:54.818252 24443 raft_consensus.cc:2454] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9bc9536c47c4335bac7dfd0cc9651bb in term 0.
I20260929 16:30:54.818678 24647 leader_election.cc:302] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49285128f79f43a2a889e7b5d16c1d72, c9bc9536c47c4335bac7dfd0cc9651bb; no voters:
I20260929 16:30:54.818920 24781 raft_consensus.cc:2781] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:30:54.818995 24781 raft_consensus.cc:482] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:30:54.819036 24781 raft_consensus.cc:3037] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:54.820007 24781 raft_consensus.cc:504] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38c8e537f8546699e6cdc4750f7603d" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44619 } } peers { permanent_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42937 } } peers { permanent_uuid: "49285128f79f43a2a889e7b5d16c1d72" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32887 } }
I20260929 16:30:54.820295 24781 leader_election.cc:288] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [CANDIDATE]: Term 1 election: Requested vote from peers b38c8e537f8546699e6cdc4750f7603d (127.23.44.1:44619), 49285128f79f43a2a889e7b5d16c1d72 (127.23.44.2:32887)
I20260929 16:30:54.820611 24577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3195d35b4e9e4508a5d514141d2b0e7e" candidate_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49285128f79f43a2a889e7b5d16c1d72"
I20260929 16:30:54.820611 24443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3195d35b4e9e4508a5d514141d2b0e7e" candidate_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38c8e537f8546699e6cdc4750f7603d"
I20260929 16:30:54.820722 24577 raft_consensus.cc:3037] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:54.820722 24443 raft_consensus.cc:3037] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:54.821920 24577 raft_consensus.cc:2454] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9bc9536c47c4335bac7dfd0cc9651bb in term 1.
I20260929 16:30:54.821939 24443 raft_consensus.cc:2454] T 3195d35b4e9e4508a5d514141d2b0e7e P b38c8e537f8546699e6cdc4750f7603d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9bc9536c47c4335bac7dfd0cc9651bb in term 1.
I20260929 16:30:54.822261 24645 leader_election.cc:302] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b38c8e537f8546699e6cdc4750f7603d, c9bc9536c47c4335bac7dfd0cc9651bb; no voters:
I20260929 16:30:54.822458 24781 raft_consensus.cc:2781] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:54.822697 24781 raft_consensus.cc:686] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [term 1 LEADER]: Becoming Leader. State: Replica: c9bc9536c47c4335bac7dfd0cc9651bb, State: Running, Role: LEADER
I20260929 16:30:54.823015 24781 consensus_queue.cc:237] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38c8e537f8546699e6cdc4750f7603d" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44619 } } peers { permanent_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42937 } } peers { permanent_uuid: "49285128f79f43a2a889e7b5d16c1d72" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32887 } }
I20260929 16:30:54.826320 24298 catalog_manager.cc:5495] T 3195d35b4e9e4508a5d514141d2b0e7e P c9bc9536c47c4335bac7dfd0cc9651bb reported cstate change: term changed from 0 to 1, leader changed from <none> to c9bc9536c47c4335bac7dfd0cc9651bb (127.23.44.3). New cstate: current_term: 1 leader_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38c8e537f8546699e6cdc4750f7603d" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9bc9536c47c4335bac7dfd0cc9651bb" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49285128f79f43a2a889e7b5d16c1d72" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32887 } health_report { overall_health: UNKNOWN } } }
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:30:54.841993 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.833744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.23.44.1:44619)
0929 16:30:54.833861 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:54.833865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:54.834042 (+ 177us) server_negotiation.cc:407] Connection header received
0929 16:30:54.834197 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:54.834201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:54.834250 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:54.834332 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:54.835051 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.835529 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.836407 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.836615 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.836724 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.839117 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:54.839125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:54.839136 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:54.839157 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:54.840565 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.840954 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.840958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.840960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.841041 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.841293 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.841296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.841302 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.841507 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:54.841648 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:54.841804 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 16:30:54.841889 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699454, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 16:30:54.854615 24624 tablet.cc:2369] T 3195d35b4e9e4508a5d514141d2b0e7e P 49285128f79f43a2a889e7b5d16c1d72: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699454, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:30:54.866781 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.859012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.23.44.1:44619)
0929 16:30:54.859142 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:54.859147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:54.859306 (+ 159us) server_negotiation.cc:407] Connection header received
0929 16:30:54.859409 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:54.859411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:54.859449 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:54.859516 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:54.860171 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.860649 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.861551 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.861754 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.861824 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.864052 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:54.864054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:54.864056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:54.864076 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:54.865313 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.865743 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.865745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.865747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.865818 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.866032 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.866034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.866035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.866174 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:54.866302 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:54.866490 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 16:30:54.866578 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:30:54.890388 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:54.882737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.23.44.1:44619)
0929 16:30:54.882863 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:54.882868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:54.883014 (+ 146us) server_negotiation.cc:407] Connection header received
0929 16:30:54.883101 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:54.883103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:54.883139 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:54.883206 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:54.883876 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.884391 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.885188 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:54.885466 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:54.885542 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:54.887821 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:54.887824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:54.887825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:54.887845 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:54.889046 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.889490 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.889492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.889493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.889560 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:54.889768 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:54.889769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:54.889770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:54.889892 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:54.889977 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:54.890128 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 16:30:54.890210 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 16:30:54.891652 24403 tablet_service.cc:2908] Rejecting scan request for tablet 3195d35b4e9e4508a5d514141d2b0e7e: Uninitialized: clean time has not yet been initialized
I20260929 16:30:54.893719 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 24360
I20260929 16:30:54.901868 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 24494
I20260929 16:30:54.906674 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 24628
I20260929 16:30:54.911983 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 24266
2026-09-29T16:30:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2345 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24798](info): set up 2 sockets
Sep 29 16:30:54 dist-test-slave-mlmz krb5kdc[24798](info): commencing operation
krb5kdc: starting...
W20260929 16:30:55.940927 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:30:56 dist-test-slave-mlmz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:30:56.059945 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:40333
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:40333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:56.162863 24811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:56.163076 24811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:56.163113 24811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:56.167344 24811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:56.167409 24811 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:30:56.167443 24811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:56.167493 24811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:56.167537 24811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:56.167577 24811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:56.172756 24811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:40333
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:40333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:56.173888 24811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:56.174860 24811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:56.183192 24817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:56.183161 24816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:56.183156 24819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:56.183332 24811 server_base.cc:1034] running on GCE node
I20260929 16:30:56.183974 24811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:56.185074 24811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:56.186264 24811 hybrid_clock.cc:648] HybridClock initialized: now 1790699456186259 us; error 53 us; skew 500 ppm
Sep 29 16:30:56 dist-test-slave-mlmz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:56.193537 24811 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:30:56.194398 24811 webserver.cc:460] Webserver started at http://127.23.44.62:37479/ using document root <none> and password file <none>
I20260929 16:30:56.194943 24811 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:56.195075 24811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:56.195302 24811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:56.197300 24811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "b608dfec40034fddaa8043ed3012577f"
format_stamp: "Formatted at 2026-09-29 16:30:56 on dist-test-slave-mlmz"
server_key: "2e142914eed11124e4925d3f7c9efd2f"
server_key_iv: "4e63b61e8e8b521367aa7480c16c5484"
server_key_version: "encryptionkey@0"
I20260929 16:30:56.197849 24811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "b608dfec40034fddaa8043ed3012577f"
format_stamp: "Formatted at 2026-09-29 16:30:56 on dist-test-slave-mlmz"
server_key: "2e142914eed11124e4925d3f7c9efd2f"
server_key_iv: "4e63b61e8e8b521367aa7480c16c5484"
server_key_version: "encryptionkey@0"
I20260929 16:30:56.201416 24811 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:30:56.203723 24826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:56.204944 24811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:30:56.205103 24811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "b608dfec40034fddaa8043ed3012577f"
format_stamp: "Formatted at 2026-09-29 16:30:56 on dist-test-slave-mlmz"
server_key: "2e142914eed11124e4925d3f7c9efd2f"
server_key_iv: "4e63b61e8e8b521367aa7480c16c5484"
server_key_version: "encryptionkey@0"
I20260929 16:30:56.205212 24811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:56.217831 24811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:56.220856 24811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:56.221050 24811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:56.228483 24811 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:40333
I20260929 16:30:56.228493 24888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:40333 every 8 connection(s)
I20260929 16:30:56.229367 24811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:56.231910 24889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:56.235179 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 24811
I20260929 16:30:56.235323 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:30:56.235576 23728 external_mini_cluster.cc:1428] Setting key 043e033ec4fb3b0eceb8771556b4d705
I20260929 16:30:56.237412 24889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f: Bootstrap starting.
I20260929 16:30:56.239413 24889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:56.240098 24889 log.cc:824] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:56.241896 24889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f: No bootstrap required, opened a new log
Sep 29 16:30:56 dist-test-slave-mlmz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:56.245851 24889 raft_consensus.cc:348] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b608dfec40034fddaa8043ed3012577f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40333 } }
I20260929 16:30:56.246105 24889 raft_consensus.cc:374] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:56.246167 24889 raft_consensus.cc:729] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b608dfec40034fddaa8043ed3012577f, State: Initialized, Role: FOLLOWER
I20260929 16:30:56.246639 24889 consensus_queue.cc:260] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b608dfec40034fddaa8043ed3012577f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40333 } }
I20260929 16:30:56.246781 24889 raft_consensus.cc:388] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:56.246865 24889 raft_consensus.cc:482] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:56.246961 24889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:56.247982 24889 raft_consensus.cc:504] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b608dfec40034fddaa8043ed3012577f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40333 } }
I20260929 16:30:56.248358 24889 leader_election.cc:302] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b608dfec40034fddaa8043ed3012577f; no voters:
I20260929 16:30:56.248659 24889 leader_election.cc:288] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:56.248749 24894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:56.248970 24894 raft_consensus.cc:686] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [term 1 LEADER]: Becoming Leader. State: Replica: b608dfec40034fddaa8043ed3012577f, State: Running, Role: LEADER
I20260929 16:30:56.249274 24894 consensus_queue.cc:237] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b608dfec40034fddaa8043ed3012577f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40333 } }
I20260929 16:30:56.249655 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.236848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.23.44.62:40333)
0929 16:30:56.237298 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:56.237320 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:56.237355 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:30:56.237968 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:56.237984 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:56.238302 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:56.238620 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:56.239384 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.240412 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.241285 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.241805 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.241974 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.244529 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:56.244536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:56.244554 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:56.244583 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:56.246520 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.247028 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.247034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.247041 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.247184 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.247468 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.247471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.247472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.247977 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:56.248171 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:56.248444 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 16:30:56.248745 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
I20260929 16:30:56.249842 24889 sys_catalog.cc:564] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:56.250727 24895 sys_catalog.cc:455] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b608dfec40034fddaa8043ed3012577f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b608dfec40034fddaa8043ed3012577f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40333 } } }
I20260929 16:30:56.250838 24895 sys_catalog.cc:458] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:56.251139 24896 sys_catalog.cc:455] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b608dfec40034fddaa8043ed3012577f. Latest consensus state: current_term: 1 leader_uuid: "b608dfec40034fddaa8043ed3012577f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b608dfec40034fddaa8043ed3012577f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40333 } } }
I20260929 16:30:56.251276 24896 sys_catalog.cc:458] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f [sys.catalog]: This master's current role is: LEADER
W20260929 16:30:56.254317 24910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:30:56.254407 24910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:30:56.254487 24903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:56.255144 24903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:56.259779 24903 catalog_manager.cc:1347] Generated new cluster ID: 76e616481f9c463783b8a200dcad346c
I20260929 16:30:56.259837 24903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:56.274184 24903 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:56.275295 24903 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:56.286629 24903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f: Generated new TSK 0
I20260929 16:30:56.287344 24903 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:30:56.316745 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:40333
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:56.422154 24915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:56.422369 24915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:56.422437 24915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:56.426378 24915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:56.426426 24915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:56.426462 24915 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:30:56.426537 24915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:30:56.431490 24915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.44.62:40333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:56.432376 24915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:56.433357 24915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:56.442075 24921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:56.442162 24923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:56.442122 24920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:56.442252 24915 server_base.cc:1034] running on GCE node
I20260929 16:30:56.442777 24915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:56.443357 24915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:56.444533 24915 hybrid_clock.cc:648] HybridClock initialized: now 1790699456444513 us; error 30 us; skew 500 ppm
Sep 29 16:30:56 dist-test-slave-mlmz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:56.447279 24915 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:30:56.448108 24915 webserver.cc:460] Webserver started at http://127.23.44.1:41053/ using document root <none> and password file <none>
I20260929 16:30:56.448634 24915 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:56.448693 24915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:56.448889 24915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:56.451047 24915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "1ce880f193034bce9414b90945a310e3"
format_stamp: "Formatted at 2026-09-29 16:30:56 on dist-test-slave-mlmz"
server_key: "b8604c10e8a65fd70ad2851026b0de74"
server_key_iv: "97908a17ef47b8e574379a6d85b11d36"
server_key_version: "encryptionkey@0"
I20260929 16:30:56.451633 24915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "1ce880f193034bce9414b90945a310e3"
format_stamp: "Formatted at 2026-09-29 16:30:56 on dist-test-slave-mlmz"
server_key: "b8604c10e8a65fd70ad2851026b0de74"
server_key_iv: "97908a17ef47b8e574379a6d85b11d36"
server_key_version: "encryptionkey@0"
I20260929 16:30:56.455614 24915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 16:30:56.458081 24930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:56.459254 24915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:30:56.459373 24915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "1ce880f193034bce9414b90945a310e3"
format_stamp: "Formatted at 2026-09-29 16:30:56 on dist-test-slave-mlmz"
server_key: "b8604c10e8a65fd70ad2851026b0de74"
server_key_iv: "97908a17ef47b8e574379a6d85b11d36"
server_key_version: "encryptionkey@0"
I20260929 16:30:56.459476 24915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:56.486275 24915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:56.489216 24915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:56.489456 24915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:56.490888 24915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:56.490938 24915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:56.491004 24915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:56.491039 24915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:30:56 dist-test-slave-mlmz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:56.502878 24915 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:34841
I20260929 16:30:56.503119 25045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:34841 every 8 connection(s)
I20260929 16:30:56.503824 24915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:56.505678 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.491750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:52861 (local address 127.23.44.62:40333)
0929 16:30:56.491906 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:56.491912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:56.492877 (+ 965us) server_negotiation.cc:407] Connection header received
0929 16:30:56.494254 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:56.494259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:56.494326 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:56.494423 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:56.495662 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.496491 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.497726 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.497966 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.498075 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.501850 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:56.501853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:56.501856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:56.501882 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:56.503537 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.504067 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.504070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.504072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.504150 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.504596 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.504600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.504602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.504842 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:56.504932 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:56.505457 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 16:30:56.505558 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:30:56.506350 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.492179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40333 (local address 127.23.44.1:52861)
0929 16:30:56.492697 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:56.492756 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:56.493986 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:56.494646 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:56.494658 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:56.494997 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:56.495500 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.495513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.496601 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.496606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.497605 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.497614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.498122 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.498126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.498283 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.498886 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:56.498915 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:56.501692 (+ 2777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:56.503646 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:56.503651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:56.503664 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:56.503957 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:56.504238 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:56.504242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:56.504244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:56.504410 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:56.504943 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:56.504948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:56.505188 (+ 240us) client_negotiation.cc:769] Sending connection context
0929 16:30:56.505380 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 16:30:56.505609 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260929 16:30:56.507534 25046 heartbeater.cc:347] Connected to a master server at 127.23.44.62:40333
I20260929 16:30:56.507692 25046 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:56.508220 25046 heartbeater.cc:511] Master 127.23.44.62:40333 requested a full tablet report, sending...
I20260929 16:30:56.509855 24842 ts_manager.cc:194] Registered new tserver with Master: 1ce880f193034bce9414b90945a310e3 (127.23.44.1:34841)
I20260929 16:30:56.511096 24842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:52861
I20260929 16:30:56.512472 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 24915
I20260929 16:30:56.512598 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:30:56.512871 23728 external_mini_cluster.cc:1428] Setting key 924a663ac28c75fd20f8af3a0c9af45e
I20260929 16:30:56.517243 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.509654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:40541 (local address 127.23.44.62:40333)
0929 16:30:56.509777 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:56.509782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:56.510012 (+ 230us) server_negotiation.cc:407] Connection header received
0929 16:30:56.510055 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:56.510058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:56.510099 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:56.510174 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:56.510635 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.511101 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.512022 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.512324 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.512428 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.513961 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:56.513964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:56.513967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:56.513989 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:56.515575 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.516087 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.516090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.516092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.516173 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.516472 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.516474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.516475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.516634 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:56.516706 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:56.517051 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 16:30:56.517143 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:30:56.517616 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.509599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40333 (local address 127.23.44.1:40541)
0929 16:30:56.509879 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:56.509895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:56.509992 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:56.510211 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:56.510214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:56.510396 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:56.510501 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.510507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.511230 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.511234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.511916 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.511924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.512577 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.512581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.512709 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.513248 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:56.513288 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:56.513822 (+ 534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:56.515698 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:56.515701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:56.515703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:56.515967 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:56.516264 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:56.516270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:56.516272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:56.516380 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:56.516731 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:56.516735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:56.516849 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 16:30:56.516968 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:30:56.517149 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:30:56.539758 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:40333
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:56.646498 25055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:56.646711 25055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:56.646751 25055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:56.650616 25055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:56.650664 25055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:56.650722 25055 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:30:56.650789 25055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:30:56.655860 25055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.44.62:40333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:56.656682 25055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:56.657545 25055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:56.666921 25060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:56.666961 25061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:56.666944 25063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:56.667237 25055 server_base.cc:1034] running on GCE node
I20260929 16:30:56.667634 25055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:56.668192 25055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:56.669332 25055 hybrid_clock.cc:648] HybridClock initialized: now 1790699456669313 us; error 31 us; skew 500 ppm
Sep 29 16:30:56 dist-test-slave-mlmz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:56.671927 25055 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:30:56.672741 25055 webserver.cc:460] Webserver started at http://127.23.44.2:33585/ using document root <none> and password file <none>
I20260929 16:30:56.673246 25055 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:56.673368 25055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:56.673589 25055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:56.675477 25055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "88f8b7bd2b43430eb4b9ad80c321822d"
format_stamp: "Formatted at 2026-09-29 16:30:56 on dist-test-slave-mlmz"
server_key: "d4b039186d0953801081cc77308a364e"
server_key_iv: "8fbd028a73722a458ee5bc6f58e13028"
server_key_version: "encryptionkey@0"
I20260929 16:30:56.675947 25055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "88f8b7bd2b43430eb4b9ad80c321822d"
format_stamp: "Formatted at 2026-09-29 16:30:56 on dist-test-slave-mlmz"
server_key: "d4b039186d0953801081cc77308a364e"
server_key_iv: "8fbd028a73722a458ee5bc6f58e13028"
server_key_version: "encryptionkey@0"
I20260929 16:30:56.679279 25055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 16:30:56.681550 25070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:56.682674 25055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:30:56.682807 25055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "88f8b7bd2b43430eb4b9ad80c321822d"
format_stamp: "Formatted at 2026-09-29 16:30:56 on dist-test-slave-mlmz"
server_key: "d4b039186d0953801081cc77308a364e"
server_key_iv: "8fbd028a73722a458ee5bc6f58e13028"
server_key_version: "encryptionkey@0"
I20260929 16:30:56.682910 25055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:56.706591 25055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:56.709967 25055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:56.710180 25055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:56.711534 25055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:56.711607 25055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:56.711684 25055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:56.711736 25055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:30:56 dist-test-slave-mlmz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:56.722698 25055 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:34603
I20260929 16:30:56.723052 25185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:34603 every 8 connection(s)
I20260929 16:30:56.723632 25055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:30:56.724956 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 25055
I20260929 16:30:56.725107 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:30:56.725435 23728 external_mini_cluster.cc:1428] Setting key fe9a1332472379aa3aabe65d1aa01c64
I20260929 16:30:56.725769 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.712452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:57629 (local address 127.23.44.62:40333)
0929 16:30:56.712587 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:56.712591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:56.713672 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 16:30:56.714801 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:56.714804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:56.714866 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:56.714952 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:56.716061 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.716645 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.717821 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.718045 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.718156 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.721789 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:56.721793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:56.721795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:56.721823 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:56.723502 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.724146 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.724150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.724151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.724254 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.724645 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.724648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.724650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.724821 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:56.724900 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:56.725541 (+ 641us) server_negotiation.cc:299] Negotiation successful
0929 16:30:56.725652 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:30:56.726697 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.712983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40333 (local address 127.23.44.2:57629)
0929 16:30:56.713529 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:56.713567 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:56.714508 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:56.715157 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:56.715163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:56.715463 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:56.715909 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.715922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.716763 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.716767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.717701 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.717710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.718188 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.718192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.718353 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.718968 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:56.718997 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:56.721629 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:56.723623 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:56.723628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:56.723641 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:56.723966 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:56.724361 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:56.724366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:56.724368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:56.724531 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:56.724931 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:56.724939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:56.725237 (+ 298us) client_negotiation.cc:769] Sending connection context
0929 16:30:56.725457 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 16:30:56.725778 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":139,"threads_started":1}
I20260929 16:30:56.727890 25186 heartbeater.cc:347] Connected to a master server at 127.23.44.62:40333
I20260929 16:30:56.728047 25186 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:56.728575 25186 heartbeater.cc:511] Master 127.23.44.62:40333 requested a full tablet report, sending...
I20260929 16:30:56.729671 24842 ts_manager.cc:194] Registered new tserver with Master: 88f8b7bd2b43430eb4b9ad80c321822d (127.23.44.2:34603)
I20260929 16:30:56.730408 24842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:57629
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:30:56.737711 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.729993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40333 (local address 127.23.44.2:43551)
0929 16:30:56.730329 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:56.730340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:56.730419 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:56.730639 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:56.730642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:56.730845 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:56.730964 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.730974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.732090 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.732094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.732969 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.732982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.733498 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.733505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.733635 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.734175 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:56.734191 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:56.734612 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:56.736312 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:56.736317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:56.736320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:56.736542 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:56.736784 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:56.736786 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:56.736788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:56.736858 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:56.737321 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:56.737324 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:56.737395 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 16:30:56.737478 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:30:56.737589 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":100,"threads_started":1}
I20260929 16:30:56.737771 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.730091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:43551 (local address 127.23.44.62:40333)
0929 16:30:56.730206 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:56.730209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:56.730416 (+ 207us) server_negotiation.cc:407] Connection header received
0929 16:30:56.730483 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:56.730486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:56.730523 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:56.730595 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:56.731088 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.731939 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.733089 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.733374 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.733454 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.734744 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:56.734747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:56.734749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:56.734768 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:56.736210 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.736625 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.736628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.736629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.736705 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.736932 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.736934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.736935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.737195 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:56.737320 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:56.737533 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 16:30:56.737617 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:30:56.752592 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:40333
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:56.859392 25195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:56.859656 25195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:56.859730 25195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:56.863904 25195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:56.863978 25195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:56.864053 25195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:30:56.864120 25195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:30:56.869387 25195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.44.62:40333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:56.870363 25195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:56.871327 25195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:30:56.880558 25195 server_base.cc:1034] running on GCE node
W20260929 16:30:56.880580 25201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:56.880573 25203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:56.880627 25200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:56.881230 25195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:56.881880 25195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:56.883080 25195 hybrid_clock.cc:648] HybridClock initialized: now 1790699456883043 us; error 51 us; skew 500 ppm
Sep 29 16:30:56 dist-test-slave-mlmz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:56.885746 25195 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:30:56.886549 25195 webserver.cc:460] Webserver started at http://127.23.44.3:41583/ using document root <none> and password file <none>
I20260929 16:30:56.887060 25195 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:56.887120 25195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:56.887315 25195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:56.889225 25195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "d1100faa013c46a28b746025a18373cf"
format_stamp: "Formatted at 2026-09-29 16:30:56 on dist-test-slave-mlmz"
server_key: "1370ab465a72ce2afcb5d55fcdb331c4"
server_key_iv: "4048a9814c28442eccbd6c56ed93dfea"
server_key_version: "encryptionkey@0"
I20260929 16:30:56.889796 25195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "d1100faa013c46a28b746025a18373cf"
format_stamp: "Formatted at 2026-09-29 16:30:56 on dist-test-slave-mlmz"
server_key: "1370ab465a72ce2afcb5d55fcdb331c4"
server_key_iv: "4048a9814c28442eccbd6c56ed93dfea"
server_key_version: "encryptionkey@0"
I20260929 16:30:56.893090 25195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:56.895308 25210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:56.896497 25195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:30:56.896595 25195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "d1100faa013c46a28b746025a18373cf"
format_stamp: "Formatted at 2026-09-29 16:30:56 on dist-test-slave-mlmz"
server_key: "1370ab465a72ce2afcb5d55fcdb331c4"
server_key_iv: "4048a9814c28442eccbd6c56ed93dfea"
server_key_version: "encryptionkey@0"
I20260929 16:30:56.896747 25195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:56.925361 25195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:56.928365 25195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:56.928552 25195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:56.929994 25195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:56.930042 25195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:56.930106 25195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:56.930147 25195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:30:56 dist-test-slave-mlmz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:56.940834 25195 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:43881
I20260929 16:30:56.940894 25325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:43881 every 8 connection(s)
I20260929 16:30:56.941766 25195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:30:56.943737 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.930888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:32953 (local address 127.23.44.62:40333)
0929 16:30:56.931008 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:56.931017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:56.932027 (+ 1010us) server_negotiation.cc:407] Connection header received
0929 16:30:56.933139 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:56.933146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:56.933219 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:56.933350 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:56.934725 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.935258 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.936498 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.936781 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.936896 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.940200 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:56.940203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:56.940206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:56.940229 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:56.941778 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.942286 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.942291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.942296 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.942392 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.942796 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.942799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.942800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.943024 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:56.943106 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:56.943525 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 16:30:56.943622 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:30:56.944497 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.931423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40333 (local address 127.23.44.3:32953)
0929 16:30:56.931888 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:56.931927 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:56.932852 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:56.933549 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:56.933561 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:56.934011 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:56.934563 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.934589 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.935385 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.935390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.936368 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.936377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.936938 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.936942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.937113 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.937713 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:56.937728 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:56.940029 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:56.941896 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:56.941902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:56.941915 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:56.942190 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:56.942496 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:56.942499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:56.942501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:56.942698 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:56.943119 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:56.943124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:56.943328 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 16:30:56.943475 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 16:30:56.943706 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":135,"threads_started":1}
I20260929 16:30:56.945695 25326 heartbeater.cc:347] Connected to a master server at 127.23.44.62:40333
I20260929 16:30:56.945827 25326 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:56.946285 25326 heartbeater.cc:511] Master 127.23.44.62:40333 requested a full tablet report, sending...
I20260929 16:30:56.947252 24843 ts_manager.cc:194] Registered new tserver with Master: d1100faa013c46a28b746025a18373cf (127.23.44.3:43881)
I20260929 16:30:56.947880 24843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:32953
I20260929 16:30:56.948326 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 25195
I20260929 16:30:56.948453 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:30:56.948773 23728 external_mini_cluster.cc:1428] Setting key 395a816c7058e400d69fff75e7991bee
I20260929 16:30:56.950644 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:30:56.955369 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.947745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40333 (local address 127.23.44.3:54947)
0929 16:30:56.948082 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:56.948099 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:56.948208 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:56.948563 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:56.948566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:56.948791 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:56.948919 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.948927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.949888 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.949892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.950521 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.950530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.951075 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.951079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.951175 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.951701 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:56.951715 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:56.952062 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:56.953840 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:56.953844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:56.953846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:56.954034 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:56.954367 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:56.954370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:56.954372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:56.954451 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:56.954826 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:56.954830 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:56.954918 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 16:30:56.955016 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 16:30:56.955164 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260929 16:30:56.955958 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.947819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:54947 (local address 127.23.44.62:40333)
0929 16:30:56.947938 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:56.947945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:56.948170 (+ 225us) server_negotiation.cc:407] Connection header received
0929 16:30:56.948315 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:56.948319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:56.948367 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:56.948449 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:56.949048 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.949779 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.950633 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.950961 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.951073 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.952197 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:56.952201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:56.952203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:56.952225 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:56.953716 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.954146 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.954150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.954152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.954258 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.954554 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.954557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.954559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.954723 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:56.954825 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:56.955601 (+ 776us) server_negotiation.cc:299] Negotiation successful
0929 16:30:56.955726 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 16:30:56 dist-test-slave-mlmz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:30:56 dist-test-slave-mlmz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:56.975987 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.967715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.23.44.62:40333)
0929 16:30:56.967832 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:56.967835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:56.967847 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:30:56.967897 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:56.967900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:56.967932 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:56.968003 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:56.968751 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.969305 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.970154 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.970370 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.970451 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.972848 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:56.972851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:56.972853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:56.972874 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:56.974316 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.974809 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.974812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.974813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.974886 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:56.975202 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:56.975204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:56.975205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:56.975358 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:56.975452 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:56.975769 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 16:30:56.975873 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:30:56.978849 24843 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:30:56.981833 24843 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:30:56.995419 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.989864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:43881 (local address 127.0.0.1:59818)
0929 16:30:56.990565 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:56.990614 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:56.990806 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:56.991426 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:56.991434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:56.991455 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:56.991576 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.991583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.992728 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.992730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.993856 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.993866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.994877 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.994880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.994980 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.995017 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:30:56.995159 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 16:30:56.995279 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":82,"threads_started":1}
I20260929 16:30:56.995455 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.990013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.23.44.3:43881)
0929 16:30:56.990325 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:56.990336 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:56.990705 (+ 369us) server_negotiation.cc:407] Connection header received
0929 16:30:56.991003 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:56.991009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:56.991162 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:56.991255 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:56.991717 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.992611 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.993991 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.994766 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.994877 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.995033 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:56.995188 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 16:30:56.995276 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":114,"threads_started":1}
I20260929 16:30:56.995934 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.990054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:34603 (local address 127.0.0.1:54790)
0929 16:30:56.990780 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:56.990795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:56.990883 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:56.991934 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:56.991937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:56.991954 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:56.992051 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.992057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.993426 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.993429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.994549 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.994559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.995520 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.995524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.995630 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.995666 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:30:56.995716 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 16:30:56.995788 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260929 16:30:56.996253 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.991006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54264 (local address 127.23.44.1:34841)
0929 16:30:56.991268 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:56.991285 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:56.991306 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:30:56.991379 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:56.991386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:56.991561 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:56.991727 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:56.992164 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.993319 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.994532 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.995333 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.995452 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.995612 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:56.996005 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 16:30:56.996110 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
I20260929 16:30:56.996454 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.990543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:34841 (local address 127.0.0.1:54264)
0929 16:30:56.990821 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:56.990835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:56.990907 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:56.991868 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:56.991874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:56.991896 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:56.992005 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.992011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.993467 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.993470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.994399 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:56.994411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.995665 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.995668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:56.995761 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.995780 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 16:30:56.995841 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 16:30:56.995893 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
I20260929 16:30:56.996553 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:56.990620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.23.44.2:34603)
0929 16:30:56.991547 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:56.991558 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:56.991580 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:30:56.991650 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:56.991656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:56.991812 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:56.991953 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:56.992173 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.993319 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.994651 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:56.995396 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:56.996118 (+ 722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:56.996283 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:56.996337 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 16:30:56.996422 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":119,"threads_started":1}
I20260929 16:30:56.998684 25260 tablet_service.cc:1467] Processing CreateTablet for tablet ab9ee167992b4eb2a6c1f4e117756c58 (DEFAULT_TABLE table=test-table [id=36371445eabe4d129cce65014c72a1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:56.998925 25120 tablet_service.cc:1467] Processing CreateTablet for tablet ab9ee167992b4eb2a6c1f4e117756c58 (DEFAULT_TABLE table=test-table [id=36371445eabe4d129cce65014c72a1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:56.999238 24977 tablet_service.cc:1467] Processing CreateTablet for tablet ab9ee167992b4eb2a6c1f4e117756c58 (DEFAULT_TABLE table=test-table [id=36371445eabe4d129cce65014c72a1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:56.999903 25260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9ee167992b4eb2a6c1f4e117756c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:57.000029 25120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9ee167992b4eb2a6c1f4e117756c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:57.000227 24977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9ee167992b4eb2a6c1f4e117756c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:57.006549 25349 tablet_bootstrap.cc:492] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d: Bootstrap starting.
I20260929 16:30:57.007725 25350 tablet_bootstrap.cc:492] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3: Bootstrap starting.
I20260929 16:30:57.008147 25351 tablet_bootstrap.cc:492] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf: Bootstrap starting.
I20260929 16:30:57.008839 25349 tablet_bootstrap.cc:654] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:57.009706 25349 log.cc:824] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:57.009826 25350 tablet_bootstrap.cc:654] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:57.010152 25351 tablet_bootstrap.cc:654] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:57.010623 25350 log.cc:824] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:57.010893 25351 log.cc:824] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:57.011832 25349 tablet_bootstrap.cc:492] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d: No bootstrap required, opened a new log
I20260929 16:30:57.012028 25349 ts_tablet_manager.cc:1397] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:30:57.012450 25350 tablet_bootstrap.cc:492] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3: No bootstrap required, opened a new log
I20260929 16:30:57.012570 25351 tablet_bootstrap.cc:492] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf: No bootstrap required, opened a new log
I20260929 16:30:57.012643 25350 ts_tablet_manager.cc:1397] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:30:57.012715 25351 ts_tablet_manager.cc:1397] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:30:57.014905 25349 raft_consensus.cc:348] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } }
I20260929 16:30:57.015136 25349 raft_consensus.cc:374] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:57.015214 25349 raft_consensus.cc:729] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f8b7bd2b43430eb4b9ad80c321822d, State: Initialized, Role: FOLLOWER
I20260929 16:30:57.015632 25349 consensus_queue.cc:260] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } }
I20260929 16:30:57.016251 25186 heartbeater.cc:503] Master 127.23.44.62:40333 was elected leader, sending a full tablet report...
I20260929 16:30:57.016336 25351 raft_consensus.cc:348] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } }
I20260929 16:30:57.016562 25351 raft_consensus.cc:374] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:57.016566 25349 ts_tablet_manager.cc:1428] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:57.016660 25351 raft_consensus.cc:729] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1100faa013c46a28b746025a18373cf, State: Initialized, Role: FOLLOWER
I20260929 16:30:57.016875 25350 raft_consensus.cc:348] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } }
I20260929 16:30:57.017095 25350 raft_consensus.cc:374] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:57.017093 25351 consensus_queue.cc:260] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } }
I20260929 16:30:57.017182 25350 raft_consensus.cc:729] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce880f193034bce9414b90945a310e3, State: Initialized, Role: FOLLOWER
I20260929 16:30:57.017598 25350 consensus_queue.cc:260] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } }
I20260929 16:30:57.018080 25351 ts_tablet_manager.cc:1428] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260929 16:30:57.018503 25350 ts_tablet_manager.cc:1428] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:30:57.018651 25326 heartbeater.cc:503] Master 127.23.44.62:40333 was elected leader, sending a full tablet report...
I20260929 16:30:57.018814 25046 heartbeater.cc:503] Master 127.23.44.62:40333 was elected leader, sending a full tablet report...
W20260929 16:30:57.192695 25327 tablet.cc:2369] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:30:57.224419 25357 raft_consensus.cc:482] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:57.224645 25357 raft_consensus.cc:504] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } }
W20260929 16:30:57.224895 25187 tablet.cc:2369] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:30:57.225852 25357 leader_election.cc:288] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1100faa013c46a28b746025a18373cf (127.23.44.3:43881), 88f8b7bd2b43430eb4b9ad80c321822d (127.23.44.2:34603)
I20260929 16:30:57.230129 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:57.226190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:52211 (local address 127.23.44.3:43881)
0929 16:30:57.226367 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:57.226372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:57.226465 (+ 93us) server_negotiation.cc:407] Connection header received
0929 16:30:57.226606 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:57.226611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:57.226670 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:57.226734 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:57.227045 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.227638 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.229045 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.229555 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.229672 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:57.229747 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:57.229922 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 16:30:57.230001 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:30:57.230247 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:57.226207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:36781 (local address 127.23.44.2:34603)
0929 16:30:57.226367 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:57.226373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:57.226533 (+ 160us) server_negotiation.cc:407] Connection header received
0929 16:30:57.226672 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:57.226675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:57.226734 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:57.226810 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:57.227141 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.227823 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.229165 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.229756 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.229855 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:57.229910 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:57.230073 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 16:30:57.230141 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 16:30:57.230253 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:57.226098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:34603 (local address 127.23.44.1:36781)
0929 16:30:57.226414 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:57.226449 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:57.226573 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:57.226838 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:57.226841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:57.226856 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:57.226959 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:57.226963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.227949 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.227952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:57.229044 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:57.229054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.229856 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.229860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:57.229960 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:57.229980 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 16:30:57.230049 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:30:57.230109 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":21,"thread_start_us":60,"threads_started":1}
I20260929 16:30:57.230559 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:57.226050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:43881 (local address 127.23.44.1:52211)
0929 16:30:57.226357 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:57.226374 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:57.226478 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:57.226767 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:57.226773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:57.226794 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:57.226930 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:57.226936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.227734 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.227737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:57.228880 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:57.228890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.229679 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.229683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:57.229777 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:57.229799 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:30:57.230260 (+ 461us) client_negotiation.cc:240] Negotiation successful
0929 16:30:57.230440 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":2,"thread_start_us":98,"threads_started":1}
I20260929 16:30:57.230938 25138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9ee167992b4eb2a6c1f4e117756c58" candidate_uuid: "1ce880f193034bce9414b90945a310e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" is_pre_election: true
I20260929 16:30:57.231199 25138 raft_consensus.cc:2454] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ce880f193034bce9414b90945a310e3 in term 0.
I20260929 16:30:57.231640 24934 leader_election.cc:302] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ce880f193034bce9414b90945a310e3, 88f8b7bd2b43430eb4b9ad80c321822d; no voters:
I20260929 16:30:57.231896 25357 raft_consensus.cc:2781] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:30:57.231989 25357 raft_consensus.cc:482] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:30:57.232028 25357 raft_consensus.cc:3037] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:57.233228 25357 raft_consensus.cc:504] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } }
I20260929 16:30:57.233575 25357 leader_election.cc:288] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [CANDIDATE]: Term 1 election: Requested vote from peers d1100faa013c46a28b746025a18373cf (127.23.44.3:43881), 88f8b7bd2b43430eb4b9ad80c321822d (127.23.44.2:34603)
I20260929 16:30:57.233923 25138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9ee167992b4eb2a6c1f4e117756c58" candidate_uuid: "1ce880f193034bce9414b90945a310e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f8b7bd2b43430eb4b9ad80c321822d"
I20260929 16:30:57.234021 25138 raft_consensus.cc:3037] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:57.234994 25138 raft_consensus.cc:2454] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ce880f193034bce9414b90945a310e3 in term 1.
I20260929 16:30:57.235283 24934 leader_election.cc:302] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ce880f193034bce9414b90945a310e3, 88f8b7bd2b43430eb4b9ad80c321822d; no voters:
I20260929 16:30:57.235433 25357 raft_consensus.cc:2781] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:57.235636 25357 raft_consensus.cc:686] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 1 LEADER]: Becoming Leader. State: Replica: 1ce880f193034bce9414b90945a310e3, State: Running, Role: LEADER
I20260929 16:30:57.235939 25357 consensus_queue.cc:237] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } }
I20260929 16:30:57.236565 25279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9ee167992b4eb2a6c1f4e117756c58" candidate_uuid: "1ce880f193034bce9414b90945a310e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1100faa013c46a28b746025a18373cf"
I20260929 16:30:57.236769 25279 raft_consensus.cc:3037] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:57.236567 25280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9ee167992b4eb2a6c1f4e117756c58" candidate_uuid: "1ce880f193034bce9414b90945a310e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1100faa013c46a28b746025a18373cf" is_pre_election: true
I20260929 16:30:57.239158 25279 raft_consensus.cc:2454] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ce880f193034bce9414b90945a310e3 in term 1.
I20260929 16:30:57.239288 24842 catalog_manager.cc:5495] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ce880f193034bce9414b90945a310e3 (127.23.44.1). New cstate: current_term: 1 leader_uuid: "1ce880f193034bce9414b90945a310e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } health_report { overall_health: HEALTHY } } }
W20260929 16:30:57.255398 25047 tablet.cc:2369] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:30:57.289896 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:57.286252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54280 (local address 127.23.44.1:34841)
0929 16:30:57.286395 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:57.286400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:57.286414 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:30:57.286638 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:57.286641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:57.286703 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:57.286803 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:57.287074 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.287590 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.288468 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.288693 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.288800 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:57.289120 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:57.289496 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:57.289620 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:57.289706 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 16:30:57.289757 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:30:57.294885 25279 raft_consensus.cc:1261] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf [term 1 FOLLOWER]: Refusing update from remote peer 1ce880f193034bce9414b90945a310e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:57.294921 25138 raft_consensus.cc:1261] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d [term 1 FOLLOWER]: Refusing update from remote peer 1ce880f193034bce9414b90945a310e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:57.295490 25357 consensus_queue.cc:1035] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:57.295666 25360 consensus_queue.cc:1035] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:57.303252 25363 mvcc.cc:205] Tried to move back new op lower bound from 7334704977073811456 to 7334704976841138176. Current Snapshot: MvccSnapshot[applied={T|T < 7334704977073811456}]
I20260929 16:30:57.314319 24841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:57.314460 24841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:57.316712 24841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f: Sending DeleteTablet for 3 replicas of tablet ab9ee167992b4eb2a6c1f4e117756c58
I20260929 16:30:57.317471 25120 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9ee167992b4eb2a6c1f4e117756c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:57 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:54790
I20260929 16:30:57.317507 25260 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9ee167992b4eb2a6c1f4e117756c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:57 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:59818
I20260929 16:30:57.317916 25373 tablet_replica.cc:330] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf: stopping tablet replica
I20260929 16:30:57.318002 25372 tablet_replica.cc:330] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d: stopping tablet replica
I20260929 16:30:57.318260 25373 raft_consensus.cc:2229] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:57.318256 24977 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9ee167992b4eb2a6c1f4e117756c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:57 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:54264
I20260929 16:30:57.318338 25372 raft_consensus.cc:2229] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:57.318622 25373 raft_consensus.cc:2258] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:57.318645 25372 raft_consensus.cc:2258] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:57.318681 24841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:30:57.319227 24841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:30:57.320024 25372 ts_tablet_manager.cc:1905] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:57.320025 25373 ts_tablet_manager.cc:1905] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:57.322661 25372 ts_tablet_manager.cc:1918] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:57.322669 25373 ts_tablet_manager.cc:1918] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:57.322752 25373 log.cc:1196] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/ab9ee167992b4eb2a6c1f4e117756c58
I20260929 16:30:57.322821 25372 log.cc:1196] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/ab9ee167992b4eb2a6c1f4e117756c58
I20260929 16:30:57.323137 25373 ts_tablet_manager.cc:1939] T ab9ee167992b4eb2a6c1f4e117756c58 P d1100faa013c46a28b746025a18373cf: Deleting consensus metadata
I20260929 16:30:57.323238 25372 ts_tablet_manager.cc:1939] T ab9ee167992b4eb2a6c1f4e117756c58 P 88f8b7bd2b43430eb4b9ad80c321822d: Deleting consensus metadata
I20260929 16:30:57.324290 24830 catalog_manager.cc:4841] TS 88f8b7bd2b43430eb4b9ad80c321822d (127.23.44.2:34603): tablet ab9ee167992b4eb2a6c1f4e117756c58 (table test-table [id=36371445eabe4d129cce65014c72a1b0]) successfully deleted
I20260929 16:30:57.324470 25260 tablet_service.cc:1467] Processing CreateTablet for tablet ede1bea804614f9ba8c1a98afd7922ad (DEFAULT_TABLE table=test-table [id=99def1d02a06459e9bf68042fe1e78be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:57.324714 25260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede1bea804614f9ba8c1a98afd7922ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:57.325505 25120 tablet_service.cc:1467] Processing CreateTablet for tablet ede1bea804614f9ba8c1a98afd7922ad (DEFAULT_TABLE table=test-table [id=99def1d02a06459e9bf68042fe1e78be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:57.325728 25120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede1bea804614f9ba8c1a98afd7922ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:57.326457 25351 tablet_bootstrap.cc:492] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf: Bootstrap starting.
I20260929 16:30:57.326651 24828 catalog_manager.cc:4841] TS d1100faa013c46a28b746025a18373cf (127.23.44.3:43881): tablet ab9ee167992b4eb2a6c1f4e117756c58 (table test-table [id=36371445eabe4d129cce65014c72a1b0]) successfully deleted
I20260929 16:30:57.327194 24977 tablet_service.cc:1467] Processing CreateTablet for tablet ede1bea804614f9ba8c1a98afd7922ad (DEFAULT_TABLE table=test-table [id=99def1d02a06459e9bf68042fe1e78be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:57.327390 25351 tablet_bootstrap.cc:654] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:57.327438 24977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede1bea804614f9ba8c1a98afd7922ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:57.327963 25349 tablet_bootstrap.cc:492] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d: Bootstrap starting.
I20260929 16:30:57.328840 25349 tablet_bootstrap.cc:654] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:57.329751 25374 tablet_replica.cc:330] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3: stopping tablet replica
I20260929 16:30:57.329988 25350 tablet_bootstrap.cc:492] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3: Bootstrap starting.
I20260929 16:30:57.330070 25374 raft_consensus.cc:2229] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:30:57.330384 25351 tablet_bootstrap.cc:492] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf: No bootstrap required, opened a new log
I20260929 16:30:57.330371 25374 raft_consensus.cc:2258] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:57.330466 25351 ts_tablet_manager.cc:1397] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 16:30:57.330847 25350 tablet_bootstrap.cc:654] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:57.330912 25351 raft_consensus.cc:348] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.331022 25351 raft_consensus.cc:374] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:57.331055 25351 raft_consensus.cc:729] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1100faa013c46a28b746025a18373cf, State: Initialized, Role: FOLLOWER
I20260929 16:30:57.331193 25351 consensus_queue.cc:260] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.331475 25351 ts_tablet_manager.cc:1428] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:30:57.331668 25349 tablet_bootstrap.cc:492] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d: No bootstrap required, opened a new log
I20260929 16:30:57.331754 25349 ts_tablet_manager.cc:1397] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:30:57.332261 25349 raft_consensus.cc:348] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.332362 25349 raft_consensus.cc:374] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:57.332396 25349 raft_consensus.cc:729] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f8b7bd2b43430eb4b9ad80c321822d, State: Initialized, Role: FOLLOWER
I20260929 16:30:57.332501 25350 tablet_bootstrap.cc:492] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3: No bootstrap required, opened a new log
I20260929 16:30:57.332504 25349 consensus_queue.cc:260] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.332566 25350 ts_tablet_manager.cc:1397] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 16:30:57.332748 25349 ts_tablet_manager.cc:1428] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:30:57.333010 25350 raft_consensus.cc:348] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.333127 25350 raft_consensus.cc:374] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:57.333159 25350 raft_consensus.cc:729] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce880f193034bce9414b90945a310e3, State: Initialized, Role: FOLLOWER
I20260929 16:30:57.333287 25350 consensus_queue.cc:260] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.333501 25350 ts_tablet_manager.cc:1428] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:30:57.333808 25374 ts_tablet_manager.cc:1905] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:57.336594 25374 ts_tablet_manager.cc:1918] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:57.336714 25374 log.cc:1196] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/wal/wals/ab9ee167992b4eb2a6c1f4e117756c58
I20260929 16:30:57.337041 25374 ts_tablet_manager.cc:1939] T ab9ee167992b4eb2a6c1f4e117756c58 P 1ce880f193034bce9414b90945a310e3: Deleting consensus metadata
I20260929 16:30:57.338058 24828 catalog_manager.cc:4841] TS 1ce880f193034bce9414b90945a310e3 (127.23.44.1:34841): tablet ab9ee167992b4eb2a6c1f4e117756c58 (table test-table [id=36371445eabe4d129cce65014c72a1b0]) successfully deleted
I20260929 16:30:57.770848 25356 raft_consensus.cc:482] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:57.771015 25356 raft_consensus.cc:504] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.772197 25356 leader_election.cc:288] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ce880f193034bce9414b90945a310e3 (127.23.44.1:34841), 88f8b7bd2b43430eb4b9ad80c321822d (127.23.44.2:34603)
I20260929 16:30:57.775933 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:57.772487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:54181 (local address 127.23.44.1:34841)
0929 16:30:57.772633 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:57.772638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:57.772788 (+ 150us) server_negotiation.cc:407] Connection header received
0929 16:30:57.772974 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:57.772976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:57.773023 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:57.773108 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:57.773439 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.773936 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.774964 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.775365 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.775501 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:57.775566 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:57.775747 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 16:30:57.775811 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:30:57.775950 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:57.772381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:34841 (local address 127.23.44.3:54181)
0929 16:30:57.772700 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:57.772717 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:57.772835 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:57.773122 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:57.773125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:57.773140 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:57.773325 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:57.773332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.774053 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.774059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:57.774840 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:57.774848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.775510 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.775514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:57.775611 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:57.775634 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:30:57.775731 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 16:30:57.775806 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20260929 16:30:57.776209 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:57.772500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:34603 (local address 127.23.44.3:39521)
0929 16:30:57.772805 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:57.772818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:57.772923 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:57.773195 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:57.773197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:57.773214 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:57.773373 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:57.773379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.774239 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.774244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:57.775085 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:57.775094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.775781 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.775784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:57.775880 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:57.775907 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:30:57.775994 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:30:57.776061 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":22,"thread_start_us":71,"threads_started":1}
I20260929 16:30:57.776201 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:57.772610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:39521 (local address 127.23.44.2:34603)
0929 16:30:57.772846 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:57.772853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:57.772869 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:30:57.773049 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:57.773051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:57.773093 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:57.773193 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:57.773598 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.774121 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.775244 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.775651 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.775779 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:57.775841 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:57.776011 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 16:30:57.776085 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260929 16:30:57.776662 25138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede1bea804614f9ba8c1a98afd7922ad" candidate_uuid: "d1100faa013c46a28b746025a18373cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" is_pre_election: true
I20260929 16:30:57.776803 25138 raft_consensus.cc:2454] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1100faa013c46a28b746025a18373cf in term 0.
I20260929 16:30:57.776753 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede1bea804614f9ba8c1a98afd7922ad" candidate_uuid: "d1100faa013c46a28b746025a18373cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ce880f193034bce9414b90945a310e3" is_pre_election: true
I20260929 16:30:57.777052 25000 raft_consensus.cc:2454] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1100faa013c46a28b746025a18373cf in term 0.
I20260929 16:30:57.777227 25214 leader_election.cc:302] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88f8b7bd2b43430eb4b9ad80c321822d, d1100faa013c46a28b746025a18373cf; no voters:
I20260929 16:30:57.777542 25356 raft_consensus.cc:2781] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:30:57.777593 25356 raft_consensus.cc:482] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:30:57.777621 25356 raft_consensus.cc:3037] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:57.778663 25356 raft_consensus.cc:504] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.779044 25356 leader_election.cc:288] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [CANDIDATE]: Term 1 election: Requested vote from peers 1ce880f193034bce9414b90945a310e3 (127.23.44.1:34841), 88f8b7bd2b43430eb4b9ad80c321822d (127.23.44.2:34603)
I20260929 16:30:57.779325 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede1bea804614f9ba8c1a98afd7922ad" candidate_uuid: "d1100faa013c46a28b746025a18373cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ce880f193034bce9414b90945a310e3"
I20260929 16:30:57.779371 25138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede1bea804614f9ba8c1a98afd7922ad" candidate_uuid: "d1100faa013c46a28b746025a18373cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f8b7bd2b43430eb4b9ad80c321822d"
I20260929 16:30:57.779464 25000 raft_consensus.cc:3037] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:57.779490 25138 raft_consensus.cc:3037] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:57.780395 25000 raft_consensus.cc:2454] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1100faa013c46a28b746025a18373cf in term 1.
I20260929 16:30:57.780428 25138 raft_consensus.cc:2454] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1100faa013c46a28b746025a18373cf in term 1.
I20260929 16:30:57.780696 25214 leader_election.cc:302] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88f8b7bd2b43430eb4b9ad80c321822d, d1100faa013c46a28b746025a18373cf; no voters:
I20260929 16:30:57.780953 25356 raft_consensus.cc:2781] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:57.781157 25356 raft_consensus.cc:686] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 1 LEADER]: Becoming Leader. State: Replica: d1100faa013c46a28b746025a18373cf, State: Running, Role: LEADER
I20260929 16:30:57.781440 25356 consensus_queue.cc:237] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.783514 24842 catalog_manager.cc:5495] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf reported cstate change: term changed from 0 to 1, leader changed from <none> to d1100faa013c46a28b746025a18373cf (127.23.44.3). New cstate: current_term: 1 leader_uuid: "d1100faa013c46a28b746025a18373cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } health_report { overall_health: HEALTHY } } }
Sep 29 16:30:57 dist-test-slave-mlmz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:57.927726 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:57.918958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40333 (local address 127.0.0.1:54510)
0929 16:30:57.919264 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:57.919277 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:57.919373 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:57.919633 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:57.919635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:57.920003 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:57.920133 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:57.920139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.920974 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.920977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:57.921604 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:57.921612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.922138 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.922141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:57.922205 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:57.922765 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:57.922778 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:57.924261 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:57.926115 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:57.926123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:57.926127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:57.926357 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:57.926699 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:57.926702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:57.926703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:57.926787 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:57.927177 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:57.927183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:57.927284 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 16:30:57.927384 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:30:57.927504 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
I20260929 16:30:57.927733 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:57.919005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.23.44.62:40333)
0929 16:30:57.919302 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:57.919310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:57.919327 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:30:57.919465 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:57.919467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:57.919507 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:57.919585 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:57.920263 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.920820 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.921785 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:57.922012 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:57.922109 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:57.924407 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:57.924410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:57.924412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:57.924432 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:57.926015 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:57.926513 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:57.926516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:57.926517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:57.926590 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:57.926888 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:57.926890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:57.926891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:57.927073 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:57.927151 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:57.927479 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 16:30:57.927609 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":134,"threads_started":1}
I20260929 16:30:57.931636 24842 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:54510:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:30:57.932363 24842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:30:57.937769 25260 tablet_service.cc:1467] Processing CreateTablet for tablet 5075ea2746254e46876a64ddeafcee5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1af3b0957f5b4b39865eefbb81237478]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:30:57.937701 24977 tablet_service.cc:1467] Processing CreateTablet for tablet 5075ea2746254e46876a64ddeafcee5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1af3b0957f5b4b39865eefbb81237478]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:30:57.937702 25120 tablet_service.cc:1467] Processing CreateTablet for tablet 5075ea2746254e46876a64ddeafcee5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1af3b0957f5b4b39865eefbb81237478]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:30:57.938190 25260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5075ea2746254e46876a64ddeafcee5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:57.938268 25120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5075ea2746254e46876a64ddeafcee5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:57.938268 24977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5075ea2746254e46876a64ddeafcee5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:57.940645 25390 tablet_bootstrap.cc:492] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: Bootstrap starting.
I20260929 16:30:57.941545 25391 tablet_bootstrap.cc:492] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d: Bootstrap starting.
I20260929 16:30:57.941738 25390 tablet_bootstrap.cc:654] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:57.942498 25391 tablet_bootstrap.cc:654] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:57.943179 25390 tablet_bootstrap.cc:492] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: No bootstrap required, opened a new log
I20260929 16:30:57.943356 25390 ts_tablet_manager.cc:1397] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:30:57.943878 25391 tablet_bootstrap.cc:492] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d: No bootstrap required, opened a new log
I20260929 16:30:57.943953 25391 ts_tablet_manager.cc:1397] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:30:57.944325 25391 raft_consensus.cc:348] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.944324 25390 raft_consensus.cc:348] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.944437 25391 raft_consensus.cc:374] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:57.944458 25390 raft_consensus.cc:374] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:57.944470 25391 raft_consensus.cc:729] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f8b7bd2b43430eb4b9ad80c321822d, State: Initialized, Role: FOLLOWER
I20260929 16:30:57.944518 25390 raft_consensus.cc:729] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1100faa013c46a28b746025a18373cf, State: Initialized, Role: FOLLOWER
I20260929 16:30:57.944592 25391 consensus_queue.cc:260] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.944658 25390 consensus_queue.cc:260] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.944909 25391 ts_tablet_manager.cc:1428] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:30:57.944898 25356 tablet_replica.cc:439] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } }
I20260929 16:30:57.944952 25390 ts_tablet_manager.cc:1428] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:30:57.945020 25356 tablet_replica.cc:442] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:30:57.945312 25394 tablet_bootstrap.cc:492] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3: Bootstrap starting.
I20260929 16:30:57.946269 25394 tablet_bootstrap.cc:654] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:57.947220 25394 tablet_bootstrap.cc:492] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3: No bootstrap required, opened a new log
I20260929 16:30:57.947297 25394 ts_tablet_manager.cc:1397] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:30:57.947798 25394 raft_consensus.cc:348] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.947916 25394 raft_consensus.cc:374] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:57.947944 25394 raft_consensus.cc:729] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce880f193034bce9414b90945a310e3, State: Initialized, Role: FOLLOWER
I20260929 16:30:57.948051 25394 consensus_queue.cc:260] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:57.948390 25394 ts_tablet_manager.cc:1428] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:30:57.948849 25395 tablet_replica.cc:439] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } }
I20260929 16:30:57.948988 25395 tablet_replica.cc:442] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:30:57.949187 25397 tablet_replica.cc:439] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } }
I20260929 16:30:57.949368 25397 tablet_replica.cc:442] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:30:58.041186 25356 raft_consensus.cc:482] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:58.041323 25356 raft_consensus.cc:504] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:58.041659 25356 leader_election.cc:288] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ce880f193034bce9414b90945a310e3 (127.23.44.1:34841), 88f8b7bd2b43430eb4b9ad80c321822d (127.23.44.2:34603)
I20260929 16:30:58.042230 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075ea2746254e46876a64ddeafcee5c" candidate_uuid: "d1100faa013c46a28b746025a18373cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ce880f193034bce9414b90945a310e3" is_pre_election: true
I20260929 16:30:58.042229 25138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075ea2746254e46876a64ddeafcee5c" candidate_uuid: "d1100faa013c46a28b746025a18373cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" is_pre_election: true
I20260929 16:30:58.042382 25000 raft_consensus.cc:2454] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1100faa013c46a28b746025a18373cf in term 0.
I20260929 16:30:58.042382 25138 raft_consensus.cc:2454] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1100faa013c46a28b746025a18373cf in term 0.
I20260929 16:30:58.042659 25214 leader_election.cc:302] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88f8b7bd2b43430eb4b9ad80c321822d, d1100faa013c46a28b746025a18373cf; no voters:
I20260929 16:30:58.042833 25356 raft_consensus.cc:2781] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:30:58.042896 25356 raft_consensus.cc:482] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:30:58.042940 25356 raft_consensus.cc:3037] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:58.043910 25356 raft_consensus.cc:504] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:58.044269 25356 leader_election.cc:288] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [CANDIDATE]: Term 1 election: Requested vote from peers 1ce880f193034bce9414b90945a310e3 (127.23.44.1:34841), 88f8b7bd2b43430eb4b9ad80c321822d (127.23.44.2:34603)
I20260929 16:30:58.044602 25138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075ea2746254e46876a64ddeafcee5c" candidate_uuid: "d1100faa013c46a28b746025a18373cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f8b7bd2b43430eb4b9ad80c321822d"
I20260929 16:30:58.044611 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075ea2746254e46876a64ddeafcee5c" candidate_uuid: "d1100faa013c46a28b746025a18373cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ce880f193034bce9414b90945a310e3"
I20260929 16:30:58.044716 25000 raft_consensus.cc:3037] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:58.044746 25138 raft_consensus.cc:3037] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:58.045711 25000 raft_consensus.cc:2454] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1100faa013c46a28b746025a18373cf in term 1.
I20260929 16:30:58.045722 25138 raft_consensus.cc:2454] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1100faa013c46a28b746025a18373cf in term 1.
I20260929 16:30:58.046012 25212 leader_election.cc:302] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ce880f193034bce9414b90945a310e3, 88f8b7bd2b43430eb4b9ad80c321822d, d1100faa013c46a28b746025a18373cf; no voters:
I20260929 16:30:58.046169 25356 raft_consensus.cc:2781] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:58.046303 25356 raft_consensus.cc:686] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [term 1 LEADER]: Becoming Leader. State: Replica: d1100faa013c46a28b746025a18373cf, State: Running, Role: LEADER
I20260929 16:30:58.046442 25356 consensus_queue.cc:237] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } }
I20260929 16:30:58.047076 25381 tablet_replica.cc:439] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: TxnStatusTablet state changed. Reason: New leader d1100faa013c46a28b746025a18373cf. Latest consensus state: current_term: 1 leader_uuid: "d1100faa013c46a28b746025a18373cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } }
I20260929 16:30:58.047195 25381 tablet_replica.cc:442] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:30:58.047417 25399 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:30:58.047601 25399 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:30:58.048147 24840 catalog_manager.cc:5495] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf reported cstate change: term changed from 0 to 1, leader changed from <none> to d1100faa013c46a28b746025a18373cf (127.23.44.3). New cstate: current_term: 1 leader_uuid: "d1100faa013c46a28b746025a18373cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } health_report { overall_health: HEALTHY } } }
I20260929 16:30:58.070631 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:58.066341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:43881 (local address 127.0.0.1:59822)
0929 16:30:58.066576 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:58.066591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:58.066684 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:58.067045 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:58.067049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:58.067058 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:30:58.067161 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:58.067166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.068050 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.068054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:58.069007 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:58.069017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.069542 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.069545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:58.069633 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:58.069682 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:30:58.070317 (+ 635us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:30:58.070332 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 16:30:58.070409 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 16:30:58.070486 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 16:30:58.070806 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:58.066432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.23.44.3:43881)
0929 16:30:58.066699 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:58.066705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:58.066718 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:30:58.066809 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:58.066811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:58.066858 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:58.067010 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:58.067321 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.067931 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.069138 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.069429 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.069514 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:58.069841 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:58.070227 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:58.070311 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:58.070509 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 16:30:58.070587 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":99,"threads_started":1}
I20260929 16:30:58.073428 25138 raft_consensus.cc:1261] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d [term 1 FOLLOWER]: Refusing update from remote peer d1100faa013c46a28b746025a18373cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:58.073879 25000 raft_consensus.cc:1261] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3 [term 1 FOLLOWER]: Refusing update from remote peer d1100faa013c46a28b746025a18373cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:58.073915 25356 consensus_queue.cc:1035] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:58.074458 25356 consensus_queue.cc:1035] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:58.075536 25395 tablet_replica.cc:439] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d: TxnStatusTablet state changed. Reason: New leader d1100faa013c46a28b746025a18373cf. Latest consensus state: current_term: 1 leader_uuid: "d1100faa013c46a28b746025a18373cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } }
I20260929 16:30:58.075663 25395 tablet_replica.cc:442] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:30:58.076278 25397 tablet_replica.cc:439] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3: TxnStatusTablet state changed. Reason: New leader d1100faa013c46a28b746025a18373cf. Latest consensus state: current_term: 1 leader_uuid: "d1100faa013c46a28b746025a18373cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } }
I20260929 16:30:58.076390 25397 tablet_replica.cc:442] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:30:58.078346 25381 tablet_replica.cc:439] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1100faa013c46a28b746025a18373cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } }
I20260929 16:30:58.078452 25381 tablet_replica.cc:442] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:30:58.078776 25381 tablet_replica.cc:439] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1100faa013c46a28b746025a18373cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } }
I20260929 16:30:58.078874 25381 tablet_replica.cc:442] T 5075ea2746254e46876a64ddeafcee5c P d1100faa013c46a28b746025a18373cf: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:30:58.078895 25397 tablet_replica.cc:439] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1100faa013c46a28b746025a18373cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } }
I20260929 16:30:58.078989 25397 tablet_replica.cc:442] T 5075ea2746254e46876a64ddeafcee5c P 1ce880f193034bce9414b90945a310e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:30:58.079385 25404 mvcc.cc:205] Tried to move back new op lower bound from 7334704980264181760 to 7334704980159524864. Current Snapshot: MvccSnapshot[applied={T|T < 7334704980264181760}]
I20260929 16:30:58.079528 25395 tablet_replica.cc:439] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1100faa013c46a28b746025a18373cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 } } peers { permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 } } peers { permanent_uuid: "d1100faa013c46a28b746025a18373cf" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 43881 } } }
I20260929 16:30:58.079645 25395 tablet_replica.cc:442] T 5075ea2746254e46876a64ddeafcee5c P 88f8b7bd2b43430eb4b9ad80c321822d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:30:58.081856 25403 mvcc.cc:205] Tried to move back new op lower bound from 7334704980264181760 to 7334704980159524864. Current Snapshot: MvccSnapshot[applied={T|T < 7334704980264181760 or (T in {7334704980264181760})}]
I20260929 16:30:58.089866 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:58.086189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.23.44.3:43881)
0929 16:30:58.086324 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:58.086329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:58.086514 (+ 185us) server_negotiation.cc:407] Connection header received
0929 16:30:58.086627 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:58.086630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:58.086677 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:58.086741 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:58.087001 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.087608 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.088806 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.089032 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.089114 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:58.089406 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:58.089471 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:58.089529 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:58.089652 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 16:30:58.089726 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:30:58.100770 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:58.096838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:57641 (local address 127.23.44.3:43881)
0929 16:30:58.096955 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:58.096959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:58.097184 (+ 225us) server_negotiation.cc:407] Connection header received
0929 16:30:58.097314 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:58.097317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:58.097364 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:58.097429 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:58.098002 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.098476 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.099678 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.099892 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.100025 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:58.100366 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:58.100436 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:58.100492 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:58.100609 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 16:30:58.100669 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":48}
I20260929 16:30:58.100879 25415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:58.096723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:43881 (local address 127.23.44.3:57641)
0929 16:30:58.097095 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:58.097109 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:58.097208 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:58.097455 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:58.097458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:58.097465 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:30:58.097891 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:58.097899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.098575 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.098578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:58.099553 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:58.099563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.100021 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.100025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:58.100158 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:58.100227 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:30:58.100523 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:30:58.100529 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:30:58.100593 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:30:58.100666 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":17,"thread_start_us":102,"threads_started":1}
I20260929 16:30:58.107367 25000 raft_consensus.cc:1261] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3 [term 1 FOLLOWER]: Refusing update from remote peer d1100faa013c46a28b746025a18373cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:58.107820 25416 consensus_queue.cc:1035] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ce880f193034bce9414b90945a310e3" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 34841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:58.107928 25138 raft_consensus.cc:1261] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d [term 1 FOLLOWER]: Refusing update from remote peer d1100faa013c46a28b746025a18373cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:58.108424 25416 consensus_queue.cc:1035] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "88f8b7bd2b43430eb4b9ad80c321822d" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 34603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:58.144690 24843 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:58.144886 24843 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:58.147311 24843 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b608dfec40034fddaa8043ed3012577f: Sending DeleteTablet for 3 replicas of tablet ede1bea804614f9ba8c1a98afd7922ad
I20260929 16:30:58.147970 25120 tablet_service.cc:1514] Processing DeleteTablet for tablet ede1bea804614f9ba8c1a98afd7922ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:58 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:54790
I20260929 16:30:58.147970 24977 tablet_service.cc:1514] Processing DeleteTablet for tablet ede1bea804614f9ba8c1a98afd7922ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:58 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:54264
I20260929 16:30:58.148070 25259 tablet_service.cc:1514] Processing DeleteTablet for tablet ede1bea804614f9ba8c1a98afd7922ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:58 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:59818
I20260929 16:30:58.148375 25420 tablet_replica.cc:330] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3: stopping tablet replica
I20260929 16:30:58.148540 25422 tablet_replica.cc:330] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf: stopping tablet replica
I20260929 16:30:58.148684 25422 raft_consensus.cc:2229] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:30:58.148831 25420 raft_consensus.cc:2229] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:58.148974 25422 raft_consensus.cc:2258] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:58.149016 25420 raft_consensus.cc:2258] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:58.149670 25422 ts_tablet_manager.cc:1905] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:58.149866 25420 ts_tablet_manager.cc:1905] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:58.152586 25420 ts_tablet_manager.cc:1918] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:30:58.152685 25420 log.cc:1196] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-0/wal/wals/ede1bea804614f9ba8c1a98afd7922ad
I20260929 16:30:58.152814 25422 ts_tablet_manager.cc:1918] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:30:58.152892 25422 log.cc:1196] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/ede1bea804614f9ba8c1a98afd7922ad
I20260929 16:30:58.152982 25420 ts_tablet_manager.cc:1939] T ede1bea804614f9ba8c1a98afd7922ad P 1ce880f193034bce9414b90945a310e3: Deleting consensus metadata
I20260929 16:30:58.153173 25422 ts_tablet_manager.cc:1939] T ede1bea804614f9ba8c1a98afd7922ad P d1100faa013c46a28b746025a18373cf: Deleting consensus metadata
I20260929 16:30:58.153770 25421 tablet_replica.cc:330] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d: stopping tablet replica
I20260929 16:30:58.153761 24828 catalog_manager.cc:4841] TS 1ce880f193034bce9414b90945a310e3 (127.23.44.1:34841): tablet ede1bea804614f9ba8c1a98afd7922ad (table test-table [id=99def1d02a06459e9bf68042fe1e78be]) successfully deleted
I20260929 16:30:58.153893 25421 raft_consensus.cc:2229] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:58.154021 25421 raft_consensus.cc:2258] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:58.154445 24828 catalog_manager.cc:4841] TS d1100faa013c46a28b746025a18373cf (127.23.44.3:43881): tablet ede1bea804614f9ba8c1a98afd7922ad (table test-table [id=99def1d02a06459e9bf68042fe1e78be]) successfully deleted
I20260929 16:30:58.154721 25421 ts_tablet_manager.cc:1905] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:58.156956 25421 ts_tablet_manager.cc:1918] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:30:58.157059 25421 log.cc:1196] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/ede1bea804614f9ba8c1a98afd7922ad
I20260929 16:30:58.157387 25421 ts_tablet_manager.cc:1939] T ede1bea804614f9ba8c1a98afd7922ad P 88f8b7bd2b43430eb4b9ad80c321822d: Deleting consensus metadata
I20260929 16:30:58.158080 24830 catalog_manager.cc:4841] TS 88f8b7bd2b43430eb4b9ad80c321822d (127.23.44.2:34603): tablet ede1bea804614f9ba8c1a98afd7922ad (table test-table [id=99def1d02a06459e9bf68042fe1e78be]) successfully deleted
Sep 29 16:30:58 dist-test-slave-mlmz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:30:58.167989 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:58.154051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54284 (local address 127.23.44.1:34841)
0929 16:30:58.154170 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:58.154175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:58.157449 (+ 3274us) server_negotiation.cc:407] Connection header received
0929 16:30:58.157601 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:58.157604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:58.157652 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:58.157704 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:58.158685 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.159207 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.160344 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.160573 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.160656 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:58.164329 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:58.164348 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:58.164359 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:58.164383 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:58.166181 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:58.166614 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:58.166620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:58.166623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:58.166705 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:58.166991 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:58.166994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:58.166996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:58.167285 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:58.167498 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:58.167693 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 16:30:58.167816 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 16:30:58.168406 25040 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54284
I20260929 16:30:58.177524 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:58.170103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.23.44.62:40333)
0929 16:30:58.170221 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:58.170226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:58.170417 (+ 191us) server_negotiation.cc:407] Connection header received
0929 16:30:58.170552 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:58.170555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:58.170608 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:58.170709 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:58.171356 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.171881 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.172664 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:58.172911 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:58.173023 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:58.174426 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:58.174434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:58.174436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:58.174462 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:58.175836 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:58.176249 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:58.176253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:58.176255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:58.176343 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:58.176618 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:58.176622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:58.176624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:58.176803 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:58.176942 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:58.177173 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 16:30:58.177373 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 16:30:58.177915 24843 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54520
I20260929 16:30:58.179139 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 24915
I20260929 16:30:58.187247 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 25055
I20260929 16:30:58.194396 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 25195
I20260929 16:30:58.201128 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 24811
2026-09-29T16:30:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3289 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:58 dist-test-slave-mlmz krb5kdc[25429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:58 dist-test-slave-mlmz krb5kdc[25429](info): set up 2 sockets
Sep 29 16:30:58 dist-test-slave-mlmz krb5kdc[25429](info): commencing operation
krb5kdc: starting...
W20260929 16:30:59.264992 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:30:59 dist-test-slave-mlmz krb5kdc[25429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:30:59.382335 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39469
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:39469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:59.487673 25442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:59.487929 25442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:59.488001 25442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:59.491904 25442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:59.491983 25442 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:30:59.492015 25442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:59.492053 25442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:59.492120 25442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:59.492156 25442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:59.497146 25442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:39469
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:59.498098 25442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:59.499012 25442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:59.507076 25450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:59.507161 25442 server_base.cc:1034] running on GCE node
W20260929 16:30:59.507086 25447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:59.507187 25448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:59.507941 25442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:59.509040 25442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:59.510241 25442 hybrid_clock.cc:648] HybridClock initialized: now 1790699459510212 us; error 41 us; skew 500 ppm
Sep 29 16:30:59 dist-test-slave-mlmz krb5kdc[25429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699459, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:59.512992 25442 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:30:59.513865 25442 webserver.cc:460] Webserver started at http://127.23.44.62:34039/ using document root <none> and password file <none>
I20260929 16:30:59.514389 25442 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:59.514474 25442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:59.514690 25442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:59.516604 25442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "7fa3d5830fb0436fb5b7af02bead0e3b"
format_stamp: "Formatted at 2026-09-29 16:30:59 on dist-test-slave-mlmz"
server_key: "04ff7aeda0e4a2f47a6fa0173281455d"
server_key_iv: "1061c60cd497371b60aa94158df1c337"
server_key_version: "encryptionkey@0"
I20260929 16:30:59.517148 25442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "7fa3d5830fb0436fb5b7af02bead0e3b"
format_stamp: "Formatted at 2026-09-29 16:30:59 on dist-test-slave-mlmz"
server_key: "04ff7aeda0e4a2f47a6fa0173281455d"
server_key_iv: "1061c60cd497371b60aa94158df1c337"
server_key_version: "encryptionkey@0"
I20260929 16:30:59.520677 25442 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:30:59.523011 25457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:59.524343 25442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:30:59.524487 25442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "7fa3d5830fb0436fb5b7af02bead0e3b"
format_stamp: "Formatted at 2026-09-29 16:30:59 on dist-test-slave-mlmz"
server_key: "04ff7aeda0e4a2f47a6fa0173281455d"
server_key_iv: "1061c60cd497371b60aa94158df1c337"
server_key_version: "encryptionkey@0"
I20260929 16:30:59.524595 25442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:59.535739 25442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:59.538533 25442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:59.538717 25442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:59.546072 25442 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:39469
I20260929 16:30:59.546092 25519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:39469 every 8 connection(s)
I20260929 16:30:59.547008 25442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:59.548170 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 25442
I20260929 16:30:59.548286 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:30:59.548584 23728 external_mini_cluster.cc:1428] Setting key 2ed550c78ace88de50458a3d18ab6f77
I20260929 16:30:59.551555 25520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:59.558039 25520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b: Bootstrap starting.
I20260929 16:30:59.560751 25520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b: Neither blocks nor log segments found. Creating new log.
Sep 29 16:30:59 dist-test-slave-mlmz krb5kdc[25429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:59.561513 25520 log.cc:824] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:59.564083 25520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b: No bootstrap required, opened a new log
I20260929 16:30:59.568086 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:59.550728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.23.44.62:39469)
0929 16:30:59.551192 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:59.551210 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:59.551249 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:30:59.552063 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:59.552084 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:59.552465 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:59.553751 (+ 1286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:59.554147 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:59.555628 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:59.556963 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:59.557662 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:59.557814 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:59.561912 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:59.561918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:59.561940 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:59.561963 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:59.564828 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:59.565390 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:59.565396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:59.565402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:59.565533 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:59.565830 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:59.565833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:59.565835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:59.566315 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:59.566540 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:59.566880 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 16:30:59.567180 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":140,"threads_started":1}
I20260929 16:30:59.567945 25520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3d5830fb0436fb5b7af02bead0e3b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39469 } }
I20260929 16:30:59.568204 25520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:59.568257 25520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3d5830fb0436fb5b7af02bead0e3b, State: Initialized, Role: FOLLOWER
I20260929 16:30:59.568732 25520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3d5830fb0436fb5b7af02bead0e3b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39469 } }
I20260929 16:30:59.568917 25520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:59.569005 25520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:59.570026 25520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:59.571224 25520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3d5830fb0436fb5b7af02bead0e3b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39469 } }
I20260929 16:30:59.571580 25520 leader_election.cc:302] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fa3d5830fb0436fb5b7af02bead0e3b; no voters:
I20260929 16:30:59.571938 25520 leader_election.cc:288] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:59.572029 25525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:59.572242 25525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3d5830fb0436fb5b7af02bead0e3b, State: Running, Role: LEADER
I20260929 16:30:59.572543 25525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3d5830fb0436fb5b7af02bead0e3b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39469 } }
I20260929 16:30:59.573398 25520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:59.574803 25527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fa3d5830fb0436fb5b7af02bead0e3b. Latest consensus state: current_term: 1 leader_uuid: "7fa3d5830fb0436fb5b7af02bead0e3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3d5830fb0436fb5b7af02bead0e3b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39469 } } }
I20260929 16:30:59.574944 25527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:59.574400 25526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fa3d5830fb0436fb5b7af02bead0e3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3d5830fb0436fb5b7af02bead0e3b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39469 } } }
I20260929 16:30:59.575238 25526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:59.575310 25534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:59.578424 25534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:59.583724 25534 catalog_manager.cc:1347] Generated new cluster ID: ee79be93fc5b488e9087dde1c7e88ea5
I20260929 16:30:59.583784 25534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:59.593981 25534 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:59.595078 25534 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:59.603626 25534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b: Generated new TSK 0
I20260929 16:30:59.604259 25534 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:30:59.639592 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39469
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:59.747953 25546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:59.748186 25546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:59.748230 25546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:59.752099 25546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:59.752152 25546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:59.752190 25546 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:30:59.752204 25546 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:30:59.752235 25546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:30:59.757189 25546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.44.62:39469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:59.758116 25546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:59.759027 25546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:59.768445 25552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:59.768584 25554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:59.768451 25551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:59.768487 25546 server_base.cc:1034] running on GCE node
I20260929 16:30:59.769025 25546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:59.769708 25546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:59.770900 25546 hybrid_clock.cc:648] HybridClock initialized: now 1790699459770855 us; error 48 us; skew 500 ppm
Sep 29 16:30:59 dist-test-slave-mlmz krb5kdc[25429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699459, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:59.773906 25546 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:30:59.774858 25546 webserver.cc:460] Webserver started at http://127.23.44.1:40127/ using document root <none> and password file <none>
I20260929 16:30:59.775422 25546 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:59.775524 25546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:59.775727 25546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:59.777765 25546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "ad69a60d0e7b478b89a501ed37549bef"
format_stamp: "Formatted at 2026-09-29 16:30:59 on dist-test-slave-mlmz"
server_key: "5e4f51b0c433874f8c3f9637754a7a65"
server_key_iv: "c6cf4e4d3c89e4b5d316c5c57b76fecf"
server_key_version: "encryptionkey@0"
I20260929 16:30:59.778308 25546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "ad69a60d0e7b478b89a501ed37549bef"
format_stamp: "Formatted at 2026-09-29 16:30:59 on dist-test-slave-mlmz"
server_key: "5e4f51b0c433874f8c3f9637754a7a65"
server_key_iv: "c6cf4e4d3c89e4b5d316c5c57b76fecf"
server_key_version: "encryptionkey@0"
I20260929 16:30:59.781934 25546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:59.784271 25561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:59.785554 25546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:30:59.785688 25546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "ad69a60d0e7b478b89a501ed37549bef"
format_stamp: "Formatted at 2026-09-29 16:30:59 on dist-test-slave-mlmz"
server_key: "5e4f51b0c433874f8c3f9637754a7a65"
server_key_iv: "c6cf4e4d3c89e4b5d316c5c57b76fecf"
server_key_version: "encryptionkey@0"
I20260929 16:30:59.785796 25546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:59.798465 25546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:59.801730 25546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:59.801928 25546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:59.803320 25546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:59.803397 25546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:59.803467 25546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:59.803524 25546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:30:59 dist-test-slave-mlmz krb5kdc[25429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699459, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:30:59.814419 25546 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:46039
I20260929 16:30:59.814494 25676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:46039 every 8 connection(s)
I20260929 16:30:59.815256 25546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:59.817626 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:59.804182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:35155 (local address 127.23.44.62:39469)
0929 16:30:59.804314 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:59.804317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:59.805419 (+ 1102us) server_negotiation.cc:407] Connection header received
0929 16:30:59.806307 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:59.806311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:59.806367 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:59.806457 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:59.808135 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:59.808775 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:59.810610 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:59.810921 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:59.811075 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:59.813846 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:59.813849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:59.813851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:59.813874 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:59.815583 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:59.816201 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:59.816204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:59.816205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:59.816282 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:59.816576 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:59.816579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:59.816580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:59.816752 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:59.816826 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:59.817402 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 16:30:59.817507 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:30:59.818334 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:59.804705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39469 (local address 127.23.44.1:35155)
0929 16:30:59.805240 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:59.805323 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:59.806111 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:59.806660 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:59.806672 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:59.807216 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:59.807930 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:59.807955 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:59.808874 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:59.808878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:59.810465 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:59.810479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:59.811118 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:59.811122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:59.811288 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:59.811839 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:59.811855 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:59.813665 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:59.815694 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:59.815700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:59.815713 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:59.816075 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:59.816368 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:59.816370 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:59.816372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:59.816490 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:59.816845 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:59.816850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:59.817083 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 16:30:59.817228 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 16:30:59.817530 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":139,"threads_started":1}
I20260929 16:30:59.819631 25677 heartbeater.cc:347] Connected to a master server at 127.23.44.62:39469
I20260929 16:30:59.819840 25677 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:59.820256 25677 heartbeater.cc:511] Master 127.23.44.62:39469 requested a full tablet report, sending...
I20260929 16:30:59.821704 25474 ts_manager.cc:194] Registered new tserver with Master: ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039)
I20260929 16:30:59.822778 25474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:35155
I20260929 16:30:59.825498 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 25546
I20260929 16:30:59.825627 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:30:59.825903 23728 external_mini_cluster.cc:1428] Setting key 74657b9aee19ad65a615bc1d5f60504f
I20260929 16:30:59.830140 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:59.821880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:54521 (local address 127.23.44.62:39469)
0929 16:30:59.821992 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:59.821997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:59.823534 (+ 1537us) server_negotiation.cc:407] Connection header received
0929 16:30:59.823637 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:59.823640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:59.823674 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:59.823736 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:59.824217 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:59.824694 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:59.825615 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:59.825914 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:59.826025 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:59.827100 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:59.827103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:59.827105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:59.827123 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:59.828568 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:59.828991 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:59.828994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:59.828996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:59.829073 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:59.829390 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:59.829393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:59.829394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:59.829558 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:59.829663 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:59.829899 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 16:30:59.830010 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 16:30:59.830152 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:59.821823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39469 (local address 127.23.44.1:54521)
0929 16:30:59.823428 (+ 1605us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:59.823440 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:59.823548 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:59.823774 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:59.823777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:59.823984 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:59.824105 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:59.824112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:59.824826 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:59.824830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:59.825494 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:59.825503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:59.826034 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:59.826038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:59.826134 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:59.826635 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:59.826648 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:59.826927 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:59.828687 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:59.828693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:59.828695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:59.828888 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:59.829177 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:59.829180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:59.829182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:59.829303 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:59.829653 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:59.829656 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:59.829747 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 16:30:59.829849 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:30:59.829980 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:30:59.852339 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39469
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:59.958811 25686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:59.959038 25686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:59.959089 25686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:59.962935 25686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:59.962985 25686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:59.963023 25686 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:30:59.963037 25686 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:30:59.963073 25686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:30:59.968035 25686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.44.62:39469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:59.968928 25686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:59.969901 25686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:59.979146 25694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:59.979202 25692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:59.979368 25691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:59.979452 25686 server_base.cc:1034] running on GCE node
I20260929 16:30:59.979838 25686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:59.980477 25686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:59.981688 25686 hybrid_clock.cc:648] HybridClock initialized: now 1790699459981661 us; error 45 us; skew 500 ppm
Sep 29 16:30:59 dist-test-slave-mlmz krb5kdc[25429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699459, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:59.984341 25686 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:30:59.985101 25686 webserver.cc:460] Webserver started at http://127.23.44.2:45009/ using document root <none> and password file <none>
I20260929 16:30:59.985656 25686 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:59.985740 25686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:59.985947 25686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:59.987862 25686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "c1ba605335af44aab6b74b703aba6cdc"
format_stamp: "Formatted at 2026-09-29 16:30:59 on dist-test-slave-mlmz"
server_key: "f88befcb005cfd67d3258ca38fe25773"
server_key_iv: "deb7b50fd2538505eed43e9dc6d83b54"
server_key_version: "encryptionkey@0"
I20260929 16:30:59.988435 25686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "c1ba605335af44aab6b74b703aba6cdc"
format_stamp: "Formatted at 2026-09-29 16:30:59 on dist-test-slave-mlmz"
server_key: "f88befcb005cfd67d3258ca38fe25773"
server_key_iv: "deb7b50fd2538505eed43e9dc6d83b54"
server_key_version: "encryptionkey@0"
I20260929 16:30:59.992076 25686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:30:59.994495 25701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:59.995709 25686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:30:59.995836 25686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "c1ba605335af44aab6b74b703aba6cdc"
format_stamp: "Formatted at 2026-09-29 16:30:59 on dist-test-slave-mlmz"
server_key: "f88befcb005cfd67d3258ca38fe25773"
server_key_iv: "deb7b50fd2538505eed43e9dc6d83b54"
server_key_version: "encryptionkey@0"
I20260929 16:30:59.995940 25686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:00.012544 25686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:00.015548 25686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:00.015751 25686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:00.017203 25686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:00.017290 25686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:00.017356 25686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:00.017392 25686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:31:00 dist-test-slave-mlmz krb5kdc[25429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699459, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:00.028159 25686 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:43937
I20260929 16:31:00.029024 25686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:00.029386 25816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:43937 every 8 connection(s)
I20260929 16:31:00.030751 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.018128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:49739 (local address 127.23.44.62:39469)
0929 16:31:00.018295 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.018299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.019192 (+ 893us) server_negotiation.cc:407] Connection header received
0929 16:31:00.020416 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.020423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.020486 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.020582 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:00.021869 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.022391 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.023505 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.023778 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.023890 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.027065 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:00.027069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:00.027072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:00.027099 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:00.028669 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:00.029252 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:00.029255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:00.029294 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:00.029402 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:00.029836 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:00.029840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:00.029841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:00.030041 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:00.030142 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.030541 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.030637 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:31:00.031472 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.018514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39469 (local address 127.23.44.2:49739)
0929 16:31:00.018999 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.019057 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.020167 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.020769 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.020780 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.021277 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:00.021719 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.021731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.022523 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.022526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.023380 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.023389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.023924 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.023928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.024146 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.024711 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:00.024729 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:00.026866 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:00.028804 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:00.028809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:00.028821 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:00.029162 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:00.029560 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:00.029563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:00.029565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:00.029722 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:00.030139 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:00.030143 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:00.030338 (+ 195us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.030486 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.030705 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260929 16:31:00.032570 25817 heartbeater.cc:347] Connected to a master server at 127.23.44.62:39469
I20260929 16:31:00.032725 25817 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:00.033174 25817 heartbeater.cc:511] Master 127.23.44.62:39469 requested a full tablet report, sending...
I20260929 16:31:00.034233 25473 ts_manager.cc:194] Registered new tserver with Master: c1ba605335af44aab6b74b703aba6cdc (127.23.44.2:43937)
I20260929 16:31:00.035090 25473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:49739
I20260929 16:31:00.037835 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 25686
I20260929 16:31:00.037972 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:00.038323 23728 external_mini_cluster.cc:1428] Setting key d2a1c5e12a76d74df90fa689a5c87d59
I20260929 16:31:00.041373 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.034590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:44303 (local address 127.23.44.62:39469)
0929 16:31:00.034714 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.034718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.034836 (+ 118us) server_negotiation.cc:407] Connection header received
0929 16:31:00.034900 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.034903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.034943 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.035029 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:00.035435 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.035918 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.036719 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.036949 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.037060 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.038073 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:00.038079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:00.038082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:00.038116 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:00.039737 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:00.040197 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:00.040202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:00.040204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:00.040281 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:00.040549 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:00.040554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:00.040555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:00.040745 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:00.040848 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.041098 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.041199 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:31:00.041383 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.034524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39469 (local address 127.23.44.2:44303)
0929 16:31:00.034756 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.034768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.034842 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.035050 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.035052 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.035196 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:00.035328 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.035333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.036043 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.036046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.036606 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.036613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.037060 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.037064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.037152 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.037619 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:00.037629 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:00.037907 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:00.039868 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:00.039874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:00.039879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:00.040096 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:00.040378 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:00.040380 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:00.040382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:00.040459 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:00.040850 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:00.040853 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:00.040938 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.041048 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.041179 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:31:00.065222 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39469
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:00.175889 25826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:00.176105 25826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:00.176177 25826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:00.180119 25826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:00.180166 25826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:00.180204 25826 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:31:00.180255 25826 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:31:00.180310 25826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:00.185398 25826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.44.62:39469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:00.186304 25826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:00.187222 25826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:00.196328 25834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:00.196338 25831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:00.196337 25832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:00.196831 25826 server_base.cc:1034] running on GCE node
I20260929 16:31:00.197216 25826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:00.197870 25826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:00.199127 25826 hybrid_clock.cc:648] HybridClock initialized: now 1790699460199093 us; error 44 us; skew 500 ppm
Sep 29 16:31:00 dist-test-slave-mlmz krb5kdc[25429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699460, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:00.202132 25826 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:31:00.203027 25826 webserver.cc:460] Webserver started at http://127.23.44.3:40821/ using document root <none> and password file <none>
I20260929 16:31:00.203586 25826 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:00.203655 25826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:00.203868 25826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:00.205986 25826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "54c925bec3634962983bc3c15a783d40"
format_stamp: "Formatted at 2026-09-29 16:31:00 on dist-test-slave-mlmz"
server_key: "c01251fba4b4cc4454af6fe3c9a0a502"
server_key_iv: "9b3da9cda912125c107011b8387c1879"
server_key_version: "encryptionkey@0"
I20260929 16:31:00.206552 25826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "54c925bec3634962983bc3c15a783d40"
format_stamp: "Formatted at 2026-09-29 16:31:00 on dist-test-slave-mlmz"
server_key: "c01251fba4b4cc4454af6fe3c9a0a502"
server_key_iv: "9b3da9cda912125c107011b8387c1879"
server_key_version: "encryptionkey@0"
I20260929 16:31:00.210479 25826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:31:00.213048 25841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:00.214356 25826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:00.214509 25826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "54c925bec3634962983bc3c15a783d40"
format_stamp: "Formatted at 2026-09-29 16:31:00 on dist-test-slave-mlmz"
server_key: "c01251fba4b4cc4454af6fe3c9a0a502"
server_key_iv: "9b3da9cda912125c107011b8387c1879"
server_key_version: "encryptionkey@0"
I20260929 16:31:00.214633 25826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:00.230481 25826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:00.234563 25826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:00.234767 25826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:00.236701 25826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:00.236773 25826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:00.236848 25826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:00.236886 25826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:31:00 dist-test-slave-mlmz krb5kdc[25429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699460, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:00.249135 25826 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:39839
I20260929 16:31:00.249396 25956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:39839 every 8 connection(s)
I20260929 16:31:00.250325 25826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:00.250624 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 25826
I20260929 16:31:00.250725 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:00.250731 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.237151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:39023 (local address 127.23.44.62:39469)
0929 16:31:00.237298 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.237302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.238180 (+ 878us) server_negotiation.cc:407] Connection header received
0929 16:31:00.239123 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.239126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.239184 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.239283 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:00.240563 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.241444 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.243026 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.243372 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.243505 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.246900 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:00.246906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:00.246909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:00.246940 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:00.248536 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:00.249155 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:00.249158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:00.249160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:00.249268 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:00.249681 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:00.249685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:00.249686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:00.249889 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:00.250009 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.250448 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.250563 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:31:00.251042 23728 external_mini_cluster.cc:1428] Setting key ea387bd18e9ee66e7e8545c9e38a8f28
I20260929 16:31:00.251461 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.237594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39469 (local address 127.23.44.3:39023)
0929 16:31:00.238041 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.238089 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.238917 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.239448 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.239454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.239832 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:00.240371 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.240404 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.241605 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.241609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.242907 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.242920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.243523 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.243527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.243729 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.244328 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:00.244352 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:00.246744 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:00.248651 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:00.248656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:00.248669 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:00.249034 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:00.249401 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:00.249406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:00.249411 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:00.249572 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:00.249998 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:00.250003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:00.250225 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.250403 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.250692 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":129,"threads_started":1}
I20260929 16:31:00.255607 25957 heartbeater.cc:347] Connected to a master server at 127.23.44.62:39469
I20260929 16:31:00.255748 25957 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:00.256278 25957 heartbeater.cc:511] Master 127.23.44.62:39469 requested a full tablet report, sending...
I20260929 16:31:00.257743 25474 ts_manager.cc:194] Registered new tserver with Master: 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839)
I20260929 16:31:00.258662 25474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:39023
I20260929 16:31:00.265111 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.255593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39469 (local address 127.23.44.3:52547)
0929 16:31:00.256002 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.256021 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.256238 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.256626 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.256632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.256933 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:00.257171 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.257187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.258090 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.258094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.258664 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.258672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.260421 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.260424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.260492 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.261011 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:00.261025 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:00.261416 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:00.263219 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:00.263222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:00.263224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:00.263435 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:00.263846 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:00.263849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:00.263850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:00.263924 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:00.264433 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:00.264436 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:00.264633 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.264743 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.264874 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":86,"threads_started":1}
I20260929 16:31:00.265187 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:31:00.265215 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.255770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:52547 (local address 127.23.44.62:39469)
0929 16:31:00.255890 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.255894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.256112 (+ 218us) server_negotiation.cc:407] Connection header received
0929 16:31:00.256374 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.256377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.256421 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.256624 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:00.257372 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.257960 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.258872 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.259166 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.259274 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.261561 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:00.261564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:00.261567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:00.261597 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:00.263092 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:00.263532 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:00.263544 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:00.263545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:00.263622 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:00.264116 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:00.264119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:00.264120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:00.264340 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:00.264443 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.264834 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.264977 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 16:31:00 dist-test-slave-mlmz krb5kdc[25429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:31:00 dist-test-slave-mlmz krb5kdc[25429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:00.292744 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.283743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.23.44.62:39469)
0929 16:31:00.283873 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.283878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.283890 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:31:00.283926 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.283929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.283963 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.284064 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:00.284763 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.285341 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.286178 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.286397 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.286502 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.289284 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:00.289289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:00.289292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:00.289323 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:00.290728 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:00.291217 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:00.291224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:00.291228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:00.291336 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:00.291674 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:00.291677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:00.291679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:00.291936 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:00.292106 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.292327 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.292469 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:31:00.296185 25474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:00.301770 25474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:00.320399 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.311028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.23.44.2:43937)
0929 16:31:00.312129 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.312145 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.312175 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:31:00.312267 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.312274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.312469 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.312626 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:00.312917 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.314166 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.318838 (+ 4672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.319559 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.319627 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.319811 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.320166 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.320256 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":122,"threads_started":1}
I20260929 16:31:00.320389 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.310851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43937 (local address 127.0.0.1:44390)
0929 16:31:00.311571 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.311624 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.311763 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.312613 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.312619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.312647 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:00.312785 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.312794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.314314 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.314318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.318663 (+ 4345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.318672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.319905 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.319907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.319985 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.320013 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.320123 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.320242 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260929 16:31:00.321138 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.312353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:46039 (local address 127.0.0.1:33088)
0929 16:31:00.312878 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.312892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.313089 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.313996 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.314001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.314023 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:00.314140 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.314149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.315963 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.315969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.317722 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.317737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.320632 (+ 2895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.320639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.320759 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.320796 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.320849 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.320940 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":36,"thread_start_us":94,"threads_started":1}
I20260929 16:31:00.321892 25979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.312505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.23.44.1:46039)
0929 16:31:00.313496 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.313514 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.313539 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:31:00.313630 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.313637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.313829 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.314316 (+ 487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:00.314503 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.315815 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.317893 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.320161 (+ 2268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.321480 (+ 1319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.321640 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.321688 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.321772 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":67,"threads_started":1}
I20260929 16:31:00.323954 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.310937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:39839 (local address 127.0.0.1:60804)
0929 16:31:00.312600 (+ 1663us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.312612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.312747 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.313540 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.313543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.313560 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:00.313652 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.313658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.315846 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.315850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.320458 (+ 4608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.320471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.323559 (+ 3088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.323563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.323667 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.323690 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.323769 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.323834 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1604,"mutex_wait_us":2247,"thread_start_us":99,"threads_started":1}
I20260929 16:31:00.324010 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.311611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.23.44.3:39839)
0929 16:31:00.312296 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.312308 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.312687 (+ 379us) server_negotiation.cc:407] Connection header received
0929 16:31:00.312895 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.312901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.313188 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.313357 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:00.313778 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.315692 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.322678 (+ 6986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.323433 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.323556 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.323698 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.323791 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.323873 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":106,"threads_started":1}
I20260929 16:31:00.323623 25751 tablet_service.cc:1467] Processing CreateTablet for tablet b3dc7ed1aa174ab0806f3293ee05e1a4 (DEFAULT_TABLE table=test-table [id=a149ac8326b14ddbac6e793cf5be1d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:00.324703 25751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3dc7ed1aa174ab0806f3293ee05e1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:00.324436 25611 tablet_service.cc:1467] Processing CreateTablet for tablet b3dc7ed1aa174ab0806f3293ee05e1a4 (DEFAULT_TABLE table=test-table [id=a149ac8326b14ddbac6e793cf5be1d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:00.325156 25611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3dc7ed1aa174ab0806f3293ee05e1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:00.327961 25888 tablet_service.cc:1467] Processing CreateTablet for tablet b3dc7ed1aa174ab0806f3293ee05e1a4 (DEFAULT_TABLE table=test-table [id=a149ac8326b14ddbac6e793cf5be1d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:00.328891 25888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3dc7ed1aa174ab0806f3293ee05e1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:00.333518 25980 tablet_bootstrap.cc:492] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef: Bootstrap starting.
I20260929 16:31:00.335683 25980 tablet_bootstrap.cc:654] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:00.336930 25980 log.cc:824] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:00.337499 25981 tablet_bootstrap.cc:492] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc: Bootstrap starting.
I20260929 16:31:00.338541 25982 tablet_bootstrap.cc:492] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40: Bootstrap starting.
I20260929 16:31:00.338894 25980 tablet_bootstrap.cc:492] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef: No bootstrap required, opened a new log
I20260929 16:31:00.339092 25980 ts_tablet_manager.cc:1397] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:31:00.339901 25981 tablet_bootstrap.cc:654] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:00.340670 25982 tablet_bootstrap.cc:654] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:00.340843 25981 log.cc:824] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:00.341596 25982 log.cc:824] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:00.342957 25981 tablet_bootstrap.cc:492] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc: No bootstrap required, opened a new log
I20260929 16:31:00.343153 25981 ts_tablet_manager.cc:1397] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:31:00.343034 25980 raft_consensus.cc:348] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } }
I20260929 16:31:00.343276 25980 raft_consensus.cc:374] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:00.343338 25980 raft_consensus.cc:729] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad69a60d0e7b478b89a501ed37549bef, State: Initialized, Role: FOLLOWER
I20260929 16:31:00.343374 25982 tablet_bootstrap.cc:492] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40: No bootstrap required, opened a new log
I20260929 16:31:00.343544 25982 ts_tablet_manager.cc:1397] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:31:00.345176 25980 consensus_queue.cc:260] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } }
I20260929 16:31:00.346134 25980 ts_tablet_manager.cc:1428] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260929 16:31:00.346451 25677 heartbeater.cc:503] Master 127.23.44.62:39469 was elected leader, sending a full tablet report...
I20260929 16:31:00.347302 25982 raft_consensus.cc:348] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } }
I20260929 16:31:00.347539 25982 raft_consensus.cc:374] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:00.347600 25982 raft_consensus.cc:729] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c925bec3634962983bc3c15a783d40, State: Initialized, Role: FOLLOWER
I20260929 16:31:00.348188 25981 raft_consensus.cc:348] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } }
I20260929 16:31:00.348451 25981 raft_consensus.cc:374] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:00.348516 25981 raft_consensus.cc:729] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ba605335af44aab6b74b703aba6cdc, State: Initialized, Role: FOLLOWER
I20260929 16:31:00.348834 25982 consensus_queue.cc:260] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } }
I20260929 16:31:00.348984 25981 consensus_queue.cc:260] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } }
I20260929 16:31:00.349956 25982 ts_tablet_manager.cc:1428] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:31:00.350051 25981 ts_tablet_manager.cc:1428] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260929 16:31:00.350358 25957 heartbeater.cc:503] Master 127.23.44.62:39469 was elected leader, sending a full tablet report...
I20260929 16:31:00.350412 25817 heartbeater.cc:503] Master 127.23.44.62:39469 was elected leader, sending a full tablet report...
I20260929 16:31:00.382166 25988 raft_consensus.cc:482] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:00.382390 25988 raft_consensus.cc:504] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } }
I20260929 16:31:00.383507 25988 leader_election.cc:288] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839), ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039)
I20260929 16:31:00.387715 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.383729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:46039 (local address 127.23.44.2:58107)
0929 16:31:00.383844 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.383858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.383945 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.384318 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.384322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.384340 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:00.384481 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.384489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.385369 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.385372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.386535 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.386545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.387327 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.387330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.387408 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.387429 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.387490 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.387552 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 16:31:00.387732 25979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.383904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:58107 (local address 127.23.44.1:46039)
0929 16:31:00.384059 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.384064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.384079 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:31:00.384131 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.384134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.384203 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.384291 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:00.384616 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.385242 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.386664 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.387214 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.387316 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.387375 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.387527 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.387609 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:31:00.388115 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.383777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:54005 (local address 127.23.44.3:39839)
0929 16:31:00.383897 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.383902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.384224 (+ 322us) server_negotiation.cc:407] Connection header received
0929 16:31:00.384283 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.384285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.384329 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.384398 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:00.385045 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.385667 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.386922 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.387528 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.387624 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.387676 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.387900 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.387981 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:31:00.388315 25631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dc7ed1aa174ab0806f3293ee05e1a4" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad69a60d0e7b478b89a501ed37549bef" is_pre_election: true
I20260929 16:31:00.388470 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.383734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:39839 (local address 127.23.44.2:54005)
0929 16:31:00.384144 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.384153 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.384217 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.384415 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.384418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.384429 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:00.384903 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.384910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.385762 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.385765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.386817 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.386824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.387645 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.387648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.387743 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.387770 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.388249 (+ 479us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.388322 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":108,"thread_start_us":165,"threads_started":1}
I20260929 16:31:00.388620 25631 raft_consensus.cc:2454] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 0.
I20260929 16:31:00.389138 25910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dc7ed1aa174ab0806f3293ee05e1a4" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c925bec3634962983bc3c15a783d40" is_pre_election: true
I20260929 16:31:00.389192 25703 leader_election.cc:302] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad69a60d0e7b478b89a501ed37549bef, c1ba605335af44aab6b74b703aba6cdc; no voters:
I20260929 16:31:00.389453 25910 raft_consensus.cc:2454] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 0.
I20260929 16:31:00.389485 25988 raft_consensus.cc:2781] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:00.389581 25988 raft_consensus.cc:482] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:00.389636 25988 raft_consensus.cc:3037] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:00.390758 25988 raft_consensus.cc:504] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } }
I20260929 16:31:00.391084 25988 leader_election.cc:288] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 election: Requested vote from peers 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839), ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039)
I20260929 16:31:00.391413 25910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dc7ed1aa174ab0806f3293ee05e1a4" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c925bec3634962983bc3c15a783d40"
I20260929 16:31:00.391546 25910 raft_consensus.cc:3037] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:00.391574 25631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dc7ed1aa174ab0806f3293ee05e1a4" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad69a60d0e7b478b89a501ed37549bef"
I20260929 16:31:00.391691 25631 raft_consensus.cc:3037] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:00.392863 25631 raft_consensus.cc:2454] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 1.
I20260929 16:31:00.392855 25910 raft_consensus.cc:2454] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 1.
I20260929 16:31:00.393218 25702 leader_election.cc:302] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54c925bec3634962983bc3c15a783d40, c1ba605335af44aab6b74b703aba6cdc; no voters:
I20260929 16:31:00.393402 25988 raft_consensus.cc:2781] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:00.393632 25988 raft_consensus.cc:686] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 1 LEADER]: Becoming Leader. State: Replica: c1ba605335af44aab6b74b703aba6cdc, State: Running, Role: LEADER
I20260929 16:31:00.393958 25988 consensus_queue.cc:237] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } }
I20260929 16:31:00.396960 25474 catalog_manager.cc:5495] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc reported cstate change: term changed from 0 to 1, leader changed from <none> to c1ba605335af44aab6b74b703aba6cdc (127.23.44.2). New cstate: current_term: 1 leader_uuid: "c1ba605335af44aab6b74b703aba6cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } health_report { overall_health: UNKNOWN } } }
Sep 29 16:31:00 dist-test-slave-mlmz krb5kdc[25429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699459, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:00.446913 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.438902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.23.44.62:39469)
0929 16:31:00.439079 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.439084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.439234 (+ 150us) server_negotiation.cc:407] Connection header received
0929 16:31:00.439382 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.439386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.439430 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.439526 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:00.440153 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.440668 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.441509 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.441716 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.441823 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.443980 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:00.443984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:00.443987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:00.444014 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:00.445203 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:00.445692 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:00.445698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:00.445700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:00.445777 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:00.446056 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:00.446060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:00.446062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:00.446232 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:00.446337 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.446613 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.446773 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":88}
I20260929 16:31:00.446913 25997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.438788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39469 (local address 127.0.0.1:38952)
0929 16:31:00.439151 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.439163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.439290 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.439540 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.439543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.439931 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:00.440045 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.440051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.440772 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.440778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.441394 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.441402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.441822 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.441825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.441904 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.442444 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:00.442455 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:00.443840 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:00.445346 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:00.445354 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:00.445361 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:00.445586 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:00.445886 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:00.445890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:00.445891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:00.445969 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:00.446336 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:00.446342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:00.446448 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.446562 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.446721 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":100,"threads_started":1}
I20260929 16:31:00.450580 25474 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:38952:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:31:00.451203 25474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:00.455961 25751 tablet_service.cc:1467] Processing CreateTablet for tablet 064481762b9949ac82eb717dbf73c397 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4c66bca7eee44ce8b052ac82a11dd91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:31:00.455901 25611 tablet_service.cc:1467] Processing CreateTablet for tablet 064481762b9949ac82eb717dbf73c397 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4c66bca7eee44ce8b052ac82a11dd91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:31:00.456290 25888 tablet_service.cc:1467] Processing CreateTablet for tablet 064481762b9949ac82eb717dbf73c397 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4c66bca7eee44ce8b052ac82a11dd91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:31:00.456617 25751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064481762b9949ac82eb717dbf73c397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:00.456624 25611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064481762b9949ac82eb717dbf73c397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:00.456807 25888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064481762b9949ac82eb717dbf73c397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:00.459071 25981 tablet_bootstrap.cc:492] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc: Bootstrap starting.
I20260929 16:31:00.459573 25980 tablet_bootstrap.cc:492] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef: Bootstrap starting.
I20260929 16:31:00.460026 25981 tablet_bootstrap.cc:654] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:00.460570 25980 tablet_bootstrap.cc:654] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:00.461179 25982 tablet_bootstrap.cc:492] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: Bootstrap starting.
I20260929 16:31:00.461377 25981 tablet_bootstrap.cc:492] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc: No bootstrap required, opened a new log
I20260929 16:31:00.461459 25981 ts_tablet_manager.cc:1397] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:31:00.461776 25980 tablet_bootstrap.cc:492] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef: No bootstrap required, opened a new log
I20260929 16:31:00.461863 25980 ts_tablet_manager.cc:1397] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.001s
I20260929 16:31:00.461931 25981 raft_consensus.cc:348] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.462075 25981 raft_consensus.cc:374] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:00.462126 25981 raft_consensus.cc:729] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ba605335af44aab6b74b703aba6cdc, State: Initialized, Role: FOLLOWER
I20260929 16:31:00.462141 25982 tablet_bootstrap.cc:654] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:00.462265 25981 consensus_queue.cc:260] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.462353 25980 raft_consensus.cc:348] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.462484 25980 raft_consensus.cc:374] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:00.462515 25980 raft_consensus.cc:729] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad69a60d0e7b478b89a501ed37549bef, State: Initialized, Role: FOLLOWER
I20260929 16:31:00.462517 25988 tablet_replica.cc:439] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } }
I20260929 16:31:00.462649 25988 tablet_replica.cc:442] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:00.462622 25980 consensus_queue.cc:260] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.462885 25981 ts_tablet_manager.cc:1428] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:00.462888 25980 ts_tablet_manager.cc:1428] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:31:00.462844 25986 tablet_replica.cc:439] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } }
I20260929 16:31:00.462965 25986 tablet_replica.cc:442] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:00.463912 25982 tablet_bootstrap.cc:492] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: No bootstrap required, opened a new log
I20260929 16:31:00.464002 25982 ts_tablet_manager.cc:1397] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 16:31:00.464493 25982 raft_consensus.cc:348] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.464603 25982 raft_consensus.cc:374] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:00.464639 25982 raft_consensus.cc:729] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c925bec3634962983bc3c15a783d40, State: Initialized, Role: FOLLOWER
I20260929 16:31:00.464782 25982 consensus_queue.cc:260] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.465021 25982 ts_tablet_manager.cc:1428] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 16:31:00.465303 25987 tablet_replica.cc:439] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } }
I20260929 16:31:00.465457 25987 tablet_replica.cc:442] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 16:31:00.503054 25958 tablet.cc:2369] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:31:00.530469 25818 tablet.cc:2369] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:31:00.568420 25678 tablet.cc:2369] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:31:00.600772 25987 raft_consensus.cc:482] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:00.600893 25987 raft_consensus.cc:504] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.601958 25987 leader_election.cc:288] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039), c1ba605335af44aab6b74b703aba6cdc (127.23.44.2:43937)
I20260929 16:31:00.605618 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.602178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43937 (local address 127.23.44.3:47749)
0929 16:31:00.602298 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.602313 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.602407 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.602755 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.602758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.602777 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:00.602925 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.602930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.603677 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.603683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.604479 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.604487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.605141 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.605144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.605245 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.605287 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.605394 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.605470 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"spinlock_wait_cycles":17152}
I20260929 16:31:00.605665 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.602283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:47749 (local address 127.23.44.2:43937)
0929 16:31:00.602537 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.602550 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.602561 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:31:00.602606 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.602608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.602647 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.602741 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:00.603040 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.603568 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.604623 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.605020 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.605142 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.605203 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.605438 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.605526 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260929 16:31:00.605831 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.602180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:46039 (local address 127.23.44.3:38191)
0929 16:31:00.602690 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.602700 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.602825 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.603065 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.603068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.603083 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:00.603214 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.603218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.603955 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.603959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.604716 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.604724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.605420 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.605424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.605507 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.605527 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.605607 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.605682 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":193,"thread_start_us":106,"threads_started":1}
I20260929 16:31:00.605832 25979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.602317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:38191 (local address 127.23.44.1:46039)
0929 16:31:00.602449 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.602453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.602774 (+ 321us) server_negotiation.cc:407] Connection header received
0929 16:31:00.602924 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.602927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.602972 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.603044 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:00.603317 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.603850 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.604842 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.605229 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.605372 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.605452 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.605633 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.605710 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:31:00.606280 25771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064481762b9949ac82eb717dbf73c397" candidate_uuid: "54c925bec3634962983bc3c15a783d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1ba605335af44aab6b74b703aba6cdc" is_pre_election: true
I20260929 16:31:00.606375 25631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064481762b9949ac82eb717dbf73c397" candidate_uuid: "54c925bec3634962983bc3c15a783d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad69a60d0e7b478b89a501ed37549bef" is_pre_election: true
I20260929 16:31:00.606514 25771 raft_consensus.cc:2454] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54c925bec3634962983bc3c15a783d40 in term 0.
I20260929 16:31:00.606559 25631 raft_consensus.cc:2454] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54c925bec3634962983bc3c15a783d40 in term 0.
I20260929 16:31:00.606905 25843 leader_election.cc:302] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54c925bec3634962983bc3c15a783d40, ad69a60d0e7b478b89a501ed37549bef; no voters:
I20260929 16:31:00.607175 25987 raft_consensus.cc:2781] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:00.607254 25987 raft_consensus.cc:482] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:00.607291 25987 raft_consensus.cc:3037] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:00.608240 25987 raft_consensus.cc:504] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.608578 25987 leader_election.cc:288] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [CANDIDATE]: Term 1 election: Requested vote from peers ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039), c1ba605335af44aab6b74b703aba6cdc (127.23.44.2:43937)
I20260929 16:31:00.608863 25631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064481762b9949ac82eb717dbf73c397" candidate_uuid: "54c925bec3634962983bc3c15a783d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad69a60d0e7b478b89a501ed37549bef"
I20260929 16:31:00.609011 25631 raft_consensus.cc:3037] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:00.608968 25771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064481762b9949ac82eb717dbf73c397" candidate_uuid: "54c925bec3634962983bc3c15a783d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1ba605335af44aab6b74b703aba6cdc"
I20260929 16:31:00.609098 25771 raft_consensus.cc:3037] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:00.610092 25631 raft_consensus.cc:2454] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54c925bec3634962983bc3c15a783d40 in term 1.
I20260929 16:31:00.610193 25771 raft_consensus.cc:2454] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54c925bec3634962983bc3c15a783d40 in term 1.
I20260929 16:31:00.610421 25843 leader_election.cc:302] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 54c925bec3634962983bc3c15a783d40, ad69a60d0e7b478b89a501ed37549bef, c1ba605335af44aab6b74b703aba6cdc; no voters:
I20260929 16:31:00.610657 25987 raft_consensus.cc:2781] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:00.610814 25987 raft_consensus.cc:686] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [term 1 LEADER]: Becoming Leader. State: Replica: 54c925bec3634962983bc3c15a783d40, State: Running, Role: LEADER
I20260929 16:31:00.611120 25987 consensus_queue.cc:237] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.613250 26002 tablet_replica.cc:439] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: TxnStatusTablet state changed. Reason: New leader 54c925bec3634962983bc3c15a783d40. Latest consensus state: current_term: 1 leader_uuid: "54c925bec3634962983bc3c15a783d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } }
I20260929 16:31:00.613385 26002 tablet_replica.cc:442] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:31:00.613621 26004 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:31:00.613790 26004 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:31:00.614378 25474 catalog_manager.cc:5495] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54c925bec3634962983bc3c15a783d40 (127.23.44.3). New cstate: current_term: 1 leader_uuid: "54c925bec3634962983bc3c15a783d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } health_report { overall_health: HEALTHY } } }
I20260929 16:31:00.656148 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.652435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60808 (local address 127.23.44.3:39839)
0929 16:31:00.652583 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.652588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.652604 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:31:00.652745 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.652755 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.652808 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.652911 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:00.653175 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.653757 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.654708 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.654939 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.655055 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.655336 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:00.655729 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:00.655855 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.655967 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.656019 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:31:00.656148 25997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.652314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:39839 (local address 127.0.0.1:60808)
0929 16:31:00.652458 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.652479 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.652588 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.652922 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.652926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.652932 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:31:00.653070 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.653076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.653870 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.653877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.654592 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.654601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.655055 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.655058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.655151 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.655191 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:31:00.655833 (+ 642us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:31:00.655849 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.655922 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.656009 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 16:31:00.661348 25771 raft_consensus.cc:1261] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Refusing update from remote peer 54c925bec3634962983bc3c15a783d40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:00.661388 25631 raft_consensus.cc:1261] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Refusing update from remote peer 54c925bec3634962983bc3c15a783d40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:00.662027 26002 consensus_queue.cc:1035] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:00.662412 25987 consensus_queue.cc:1035] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:00.665145 25988 tablet_replica.cc:439] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc: TxnStatusTablet state changed. Reason: New leader 54c925bec3634962983bc3c15a783d40. Latest consensus state: current_term: 1 leader_uuid: "54c925bec3634962983bc3c15a783d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } }
I20260929 16:31:00.665287 25988 tablet_replica.cc:442] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:00.666468 25986 tablet_replica.cc:439] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef: TxnStatusTablet state changed. Reason: New leader 54c925bec3634962983bc3c15a783d40. Latest consensus state: current_term: 1 leader_uuid: "54c925bec3634962983bc3c15a783d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } }
I20260929 16:31:00.666611 25986 tablet_replica.cc:442] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:00.667687 25987 tablet_replica.cc:439] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54c925bec3634962983bc3c15a783d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } }
I20260929 16:31:00.667784 25987 tablet_replica.cc:442] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:31:00.668946 25988 tablet_replica.cc:439] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54c925bec3634962983bc3c15a783d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } }
I20260929 16:31:00.669049 25988 tablet_replica.cc:442] T 064481762b9949ac82eb717dbf73c397 P c1ba605335af44aab6b74b703aba6cdc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:00.669862 26008 mvcc.cc:205] Tried to move back new op lower bound from 7334704990862200832 to 7334704990665453568. Current Snapshot: MvccSnapshot[applied={T|T < 7334704990862200832}]
I20260929 16:31:00.670158 26002 tablet_replica.cc:439] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54c925bec3634962983bc3c15a783d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } }
I20260929 16:31:00.670254 26002 tablet_replica.cc:442] T 064481762b9949ac82eb717dbf73c397 P 54c925bec3634962983bc3c15a783d40: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:31:00.671404 25986 tablet_replica.cc:439] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54c925bec3634962983bc3c15a783d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } } }
I20260929 16:31:00.671511 25986 tablet_replica.cc:442] T 064481762b9949ac82eb717dbf73c397 P ad69a60d0e7b478b89a501ed37549bef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:00.673219 26009 mvcc.cc:205] Tried to move back new op lower bound from 7334704990862200832 to 7334704990665453568. Current Snapshot: MvccSnapshot[applied={T|T < 7334704990862200832 or (T in {7334704990862200832})}]
I20260929 16:31:00.681507 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.677986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.23.44.2:43937)
0929 16:31:00.678229 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.678235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.678250 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:31:00.678318 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.678320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.678361 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.678460 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:00.678730 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.679227 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.680109 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.680323 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.680423 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.680708 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:00.681059 (+ 351us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:00.681180 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.681280 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.681328 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260929 16:31:00.691749 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.688111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:60167 (local address 127.23.44.3:39839)
0929 16:31:00.688226 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.688238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.688367 (+ 129us) server_negotiation.cc:407] Connection header received
0929 16:31:00.688501 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.688504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.688559 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.688620 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:00.688902 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.689563 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.690774 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.690973 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.691053 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.691385 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:00.691442 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:00.691488 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.691621 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.691660 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:31:00.691815 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.687977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:39839 (local address 127.23.44.2:60167)
0929 16:31:00.688284 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.688299 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.688393 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.688650 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.688653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.688661 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:31:00.688789 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.688797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.689677 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.689681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.690665 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.690676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.691096 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.691101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.691193 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.691239 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:31:00.691516 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:31:00.691522 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.691587 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.691662 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":112,"threads_started":1}
I20260929 16:31:00.697964 25631 raft_consensus.cc:1261] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Refusing update from remote peer c1ba605335af44aab6b74b703aba6cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:00.698081 25910 raft_consensus.cc:1261] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Refusing update from remote peer c1ba605335af44aab6b74b703aba6cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:00.698487 25990 consensus_queue.cc:1035] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:00.698637 25988 consensus_queue.cc:1035] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:00.703169 26007 mvcc.cc:205] Tried to move back new op lower bound from 7334704991014375424 to 7334704989776044032. Current Snapshot: MvccSnapshot[applied={T|T < 7334704991014375424}]
I20260929 16:31:00.721509 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.717393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43937 (local address 127.23.44.3:36183)
0929 16:31:00.717533 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:00.717556 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:00.717642 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:00.717899 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:00.717903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:00.717914 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:31:00.718033 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.718058 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.719203 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.719208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.720205 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:00.720217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.720743 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.720747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:00.720866 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.720917 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:31:00.721193 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:31:00.721200 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:31:00.721296 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:31:00.721370 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 16:31:00.721519 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:00.717449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:36183 (local address 127.23.44.2:43937)
0929 16:31:00.717570 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:00.717574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:00.717605 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:31:00.717708 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:00.717711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:00.717782 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:00.717858 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:00.718311 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.719069 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.720348 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:00.720637 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:00.720714 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:00.721045 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:00.721106 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:00.721159 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:00.721324 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 16:31:00.721385 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:31:00.747875 25473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:00.748072 25473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:00.750855 25473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b: Sending DeleteTablet for 3 replicas of tablet b3dc7ed1aa174ab0806f3293ee05e1a4
I20260929 16:31:00.751523 25751 tablet_service.cc:1514] Processing DeleteTablet for tablet b3dc7ed1aa174ab0806f3293ee05e1a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:00 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:44390
I20260929 16:31:00.751750 25611 tablet_service.cc:1514] Processing DeleteTablet for tablet b3dc7ed1aa174ab0806f3293ee05e1a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:00 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:33088
I20260929 16:31:00.752100 26026 tablet_replica.cc:330] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc: stopping tablet replica
I20260929 16:31:00.752420 26026 raft_consensus.cc:2229] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:00.752471 25888 tablet_service.cc:1514] Processing DeleteTablet for tablet b3dc7ed1aa174ab0806f3293ee05e1a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:00 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:60804
I20260929 16:31:00.752879 26026 raft_consensus.cc:2258] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:00.753243 26028 tablet_replica.cc:330] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40: stopping tablet replica
I20260929 16:31:00.753616 26028 raft_consensus.cc:2229] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:00.753661 25473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 16:31:00.754004 26028 raft_consensus.cc:2258] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 16:31:00.754045 25473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:00.754518 26026 ts_tablet_manager.cc:1905] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:00.755378 26028 ts_tablet_manager.cc:1905] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:00.755996 26027 tablet_replica.cc:330] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef: stopping tablet replica
I20260929 16:31:00.756278 26027 raft_consensus.cc:2229] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:00.756487 26027 raft_consensus.cc:2258] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:00.757795 26026 ts_tablet_manager.cc:1918] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:31:00.757881 26026 log.cc:1196] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/b3dc7ed1aa174ab0806f3293ee05e1a4
I20260929 16:31:00.758208 26026 ts_tablet_manager.cc:1939] T b3dc7ed1aa174ab0806f3293ee05e1a4 P c1ba605335af44aab6b74b703aba6cdc: Deleting consensus metadata
I20260929 16:31:00.759320 25461 catalog_manager.cc:4841] TS c1ba605335af44aab6b74b703aba6cdc (127.23.44.2:43937): tablet b3dc7ed1aa174ab0806f3293ee05e1a4 (table test-table [id=a149ac8326b14ddbac6e793cf5be1d6c]) successfully deleted
I20260929 16:31:00.760529 25611 tablet_service.cc:1467] Processing CreateTablet for tablet d3dc54a94ed84dbebc1467c4acfdf760 (DEFAULT_TABLE table=test-table [id=416d39dbbdfa497e961ce30487991c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:00.760556 26027 ts_tablet_manager.cc:1905] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:00.761309 25751 tablet_service.cc:1467] Processing CreateTablet for tablet d3dc54a94ed84dbebc1467c4acfdf760 (DEFAULT_TABLE table=test-table [id=416d39dbbdfa497e961ce30487991c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:00.761536 25751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3dc54a94ed84dbebc1467c4acfdf760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:00.761483 25888 tablet_service.cc:1467] Processing CreateTablet for tablet d3dc54a94ed84dbebc1467c4acfdf760 (DEFAULT_TABLE table=test-table [id=416d39dbbdfa497e961ce30487991c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:00.761713 25888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3dc54a94ed84dbebc1467c4acfdf760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:00.763711 25981 tablet_bootstrap.cc:492] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc: Bootstrap starting.
I20260929 16:31:00.763917 25982 tablet_bootstrap.cc:492] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40: Bootstrap starting.
I20260929 16:31:00.764612 25981 tablet_bootstrap.cc:654] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:00.764652 25611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3dc54a94ed84dbebc1467c4acfdf760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:00.764864 25982 tablet_bootstrap.cc:654] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:00.767032 25980 tablet_bootstrap.cc:492] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef: Bootstrap starting.
I20260929 16:31:00.767926 25980 tablet_bootstrap.cc:654] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:00.767917 26027 ts_tablet_manager.cc:1918] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:31:00.768486 26027 log.cc:1196] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/wal/wals/b3dc7ed1aa174ab0806f3293ee05e1a4
I20260929 16:31:00.768781 26027 ts_tablet_manager.cc:1939] T b3dc7ed1aa174ab0806f3293ee05e1a4 P ad69a60d0e7b478b89a501ed37549bef: Deleting consensus metadata
I20260929 16:31:00.769933 25459 catalog_manager.cc:4841] TS ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039): tablet b3dc7ed1aa174ab0806f3293ee05e1a4 (table test-table [id=a149ac8326b14ddbac6e793cf5be1d6c]) successfully deleted
I20260929 16:31:00.771744 26028 ts_tablet_manager.cc:1918] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:31:00.771840 26028 log.cc:1196] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/b3dc7ed1aa174ab0806f3293ee05e1a4
I20260929 16:31:00.772141 26028 ts_tablet_manager.cc:1939] T b3dc7ed1aa174ab0806f3293ee05e1a4 P 54c925bec3634962983bc3c15a783d40: Deleting consensus metadata
I20260929 16:31:00.773190 25458 catalog_manager.cc:4841] TS 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839): tablet b3dc7ed1aa174ab0806f3293ee05e1a4 (table test-table [id=a149ac8326b14ddbac6e793cf5be1d6c]) successfully deleted
I20260929 16:31:00.774294 25982 tablet_bootstrap.cc:492] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40: No bootstrap required, opened a new log
I20260929 16:31:00.774384 25982 ts_tablet_manager.cc:1397] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260929 16:31:00.774524 25981 tablet_bootstrap.cc:492] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc: No bootstrap required, opened a new log
I20260929 16:31:00.774600 25981 ts_tablet_manager.cc:1397] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260929 16:31:00.774712 25980 tablet_bootstrap.cc:492] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef: No bootstrap required, opened a new log
I20260929 16:31:00.774801 25980 ts_tablet_manager.cc:1397] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260929 16:31:00.774852 25982 raft_consensus.cc:348] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.774966 25982 raft_consensus.cc:374] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:00.775004 25982 raft_consensus.cc:729] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c925bec3634962983bc3c15a783d40, State: Initialized, Role: FOLLOWER
I20260929 16:31:00.775007 25981 raft_consensus.cc:348] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.775110 25981 raft_consensus.cc:374] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:00.775141 25981 raft_consensus.cc:729] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ba605335af44aab6b74b703aba6cdc, State: Initialized, Role: FOLLOWER
I20260929 16:31:00.775134 25982 consensus_queue.cc:260] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.775254 25981 consensus_queue.cc:260] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.775311 25980 raft_consensus.cc:348] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.775395 25982 ts_tablet_manager.cc:1428] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:00.775414 25980 raft_consensus.cc:374] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:00.775444 25980 raft_consensus.cc:729] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad69a60d0e7b478b89a501ed37549bef, State: Initialized, Role: FOLLOWER
I20260929 16:31:00.775527 25981 ts_tablet_manager.cc:1428] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:00.775553 25980 consensus_queue.cc:260] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.775779 25980 ts_tablet_manager.cc:1428] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:00.963572 25988 raft_consensus.cc:482] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:00.963716 25988 raft_consensus.cc:504] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.964275 25988 leader_election.cc:288] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039), 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839)
I20260929 16:31:00.964774 25910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3dc54a94ed84dbebc1467c4acfdf760" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c925bec3634962983bc3c15a783d40" is_pre_election: true
I20260929 16:31:00.964774 25631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3dc54a94ed84dbebc1467c4acfdf760" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad69a60d0e7b478b89a501ed37549bef" is_pre_election: true
I20260929 16:31:00.964933 25631 raft_consensus.cc:2454] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 0.
I20260929 16:31:00.964933 25910 raft_consensus.cc:2454] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 0.
I20260929 16:31:00.965287 25702 leader_election.cc:302] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54c925bec3634962983bc3c15a783d40, c1ba605335af44aab6b74b703aba6cdc; no voters:
I20260929 16:31:00.965481 25988 raft_consensus.cc:2781] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:00.965547 25988 raft_consensus.cc:482] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:00.965595 25988 raft_consensus.cc:3037] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:00.966578 25988 raft_consensus.cc:504] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.966902 25988 leader_election.cc:288] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 election: Requested vote from peers ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039), 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839)
I20260929 16:31:00.967224 25631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3dc54a94ed84dbebc1467c4acfdf760" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad69a60d0e7b478b89a501ed37549bef"
I20260929 16:31:00.967274 25910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3dc54a94ed84dbebc1467c4acfdf760" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c925bec3634962983bc3c15a783d40"
I20260929 16:31:00.967352 25631 raft_consensus.cc:3037] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:00.967375 25910 raft_consensus.cc:3037] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:00.968438 25910 raft_consensus.cc:2454] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 1.
I20260929 16:31:00.968438 25631 raft_consensus.cc:2454] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 1.
I20260929 16:31:00.968725 25702 leader_election.cc:302] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54c925bec3634962983bc3c15a783d40, c1ba605335af44aab6b74b703aba6cdc; no voters:
I20260929 16:31:00.968915 25988 raft_consensus.cc:2781] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:00.969014 25988 raft_consensus.cc:686] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 1 LEADER]: Becoming Leader. State: Replica: c1ba605335af44aab6b74b703aba6cdc, State: Running, Role: LEADER
I20260929 16:31:00.969165 25988 consensus_queue.cc:237] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:00.971033 25473 catalog_manager.cc:5495] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc reported cstate change: term changed from 0 to 1, leader changed from <none> to c1ba605335af44aab6b74b703aba6cdc (127.23.44.2). New cstate: current_term: 1 leader_uuid: "c1ba605335af44aab6b74b703aba6cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20260929 16:31:01.018532 25910 raft_consensus.cc:1261] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Refusing update from remote peer c1ba605335af44aab6b74b703aba6cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:01.018904 25631 raft_consensus.cc:1261] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Refusing update from remote peer c1ba605335af44aab6b74b703aba6cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:01.018944 25988 consensus_queue.cc:1035] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:01.019284 25990 consensus_queue.cc:1035] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:31:01.034519 26008 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 16:31:02.583379 25473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:02.583524 25473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:02.585909 25473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b: Sending DeleteTablet for 3 replicas of tablet d3dc54a94ed84dbebc1467c4acfdf760
I20260929 16:31:02.586483 25611 tablet_service.cc:1514] Processing DeleteTablet for tablet d3dc54a94ed84dbebc1467c4acfdf760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:02 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:33088
I20260929 16:31:02.586565 25751 tablet_service.cc:1514] Processing DeleteTablet for tablet d3dc54a94ed84dbebc1467c4acfdf760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:02 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:44390
I20260929 16:31:02.586694 25888 tablet_service.cc:1514] Processing DeleteTablet for tablet d3dc54a94ed84dbebc1467c4acfdf760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:02 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:60804
I20260929 16:31:02.586773 26057 tablet_replica.cc:330] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef: stopping tablet replica
I20260929 16:31:02.586892 26057 raft_consensus.cc:2229] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:02.587004 26059 tablet_replica.cc:330] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40: stopping tablet replica
I20260929 16:31:02.587044 26057 raft_consensus.cc:2258] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:02.587128 26058 tablet_replica.cc:330] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc: stopping tablet replica
I20260929 16:31:02.587245 26058 raft_consensus.cc:2229] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:02.587136 26059 raft_consensus.cc:2229] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:02.587426 26059 raft_consensus.cc:2258] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:02.587427 26058 raft_consensus.cc:2258] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:02.587817 26057 ts_tablet_manager.cc:1905] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:02.588013 26058 ts_tablet_manager.cc:1905] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:02.590344 26059 ts_tablet_manager.cc:1905] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:02.590849 26057 ts_tablet_manager.cc:1918] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:31:02.590976 26057 log.cc:1196] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/wal/wals/d3dc54a94ed84dbebc1467c4acfdf760
I20260929 16:31:02.591099 26058 ts_tablet_manager.cc:1918] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:31:02.591176 26058 log.cc:1196] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/d3dc54a94ed84dbebc1467c4acfdf760
I20260929 16:31:02.591409 26057 ts_tablet_manager.cc:1939] T d3dc54a94ed84dbebc1467c4acfdf760 P ad69a60d0e7b478b89a501ed37549bef: Deleting consensus metadata
I20260929 16:31:02.591498 26058 ts_tablet_manager.cc:1939] T d3dc54a94ed84dbebc1467c4acfdf760 P c1ba605335af44aab6b74b703aba6cdc: Deleting consensus metadata
I20260929 16:31:02.592203 25459 catalog_manager.cc:4841] TS ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039): tablet d3dc54a94ed84dbebc1467c4acfdf760 (table test-table [id=416d39dbbdfa497e961ce30487991c48]) successfully deleted
I20260929 16:31:02.592373 25461 catalog_manager.cc:4841] TS c1ba605335af44aab6b74b703aba6cdc (127.23.44.2:43937): tablet d3dc54a94ed84dbebc1467c4acfdf760 (table test-table [id=416d39dbbdfa497e961ce30487991c48]) successfully deleted
I20260929 16:31:02.592981 26059 ts_tablet_manager.cc:1918] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:31:02.593044 26059 log.cc:1196] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/d3dc54a94ed84dbebc1467c4acfdf760
I20260929 16:31:02.593348 26059 ts_tablet_manager.cc:1939] T d3dc54a94ed84dbebc1467c4acfdf760 P 54c925bec3634962983bc3c15a783d40: Deleting consensus metadata
I20260929 16:31:02.594038 25458 catalog_manager.cc:4841] TS 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839): tablet d3dc54a94ed84dbebc1467c4acfdf760 (table test-table [id=416d39dbbdfa497e961ce30487991c48]) successfully deleted
Sep 29 16:31:02 dist-test-slave-mlmz krb5kdc[25429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:31:02 dist-test-slave-mlmz krb5kdc[25429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:02.618422 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:02.605348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.23.44.62:39469)
0929 16:31:02.605618 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:02.605624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:02.605637 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:31:02.605740 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:02.605742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:02.605782 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:02.605855 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:02.606524 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:02.607066 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:02.607868 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:02.608104 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:02.608194 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:02.615302 (+ 7108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:02.615305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:02.615308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:02.615330 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:02.616725 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:02.617124 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:02.617127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:02.617129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:02.617209 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:02.617593 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:02.617596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:02.617597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:02.617784 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:02.617887 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:02.618122 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 16:31:02.618222 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
I20260929 16:31:02.621011 25473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:02.621539 25473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:02.625824 25611 tablet_service.cc:1467] Processing CreateTablet for tablet 8f43c4d7ba5441288dd16cb0d2291dd0 (DEFAULT_TABLE table=test-table [id=4f6186ca5efb48ffbe94f59a76471757]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:02.625902 25751 tablet_service.cc:1467] Processing CreateTablet for tablet 8f43c4d7ba5441288dd16cb0d2291dd0 (DEFAULT_TABLE table=test-table [id=4f6186ca5efb48ffbe94f59a76471757]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:02.626133 25611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f43c4d7ba5441288dd16cb0d2291dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:02.626078 25888 tablet_service.cc:1467] Processing CreateTablet for tablet 8f43c4d7ba5441288dd16cb0d2291dd0 (DEFAULT_TABLE table=test-table [id=4f6186ca5efb48ffbe94f59a76471757]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:02.626149 25751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f43c4d7ba5441288dd16cb0d2291dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:02.626358 25888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f43c4d7ba5441288dd16cb0d2291dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:02.628463 26068 tablet_bootstrap.cc:492] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40: Bootstrap starting.
I20260929 16:31:02.628809 26069 tablet_bootstrap.cc:492] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef: Bootstrap starting.
I20260929 16:31:02.629498 26068 tablet_bootstrap.cc:654] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:02.629901 26069 tablet_bootstrap.cc:654] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:02.630499 26070 tablet_bootstrap.cc:492] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc: Bootstrap starting.
I20260929 16:31:02.630569 26068 tablet_bootstrap.cc:492] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40: No bootstrap required, opened a new log
I20260929 16:31:02.630658 26068 ts_tablet_manager.cc:1397] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:31:02.631028 26069 tablet_bootstrap.cc:492] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef: No bootstrap required, opened a new log
I20260929 16:31:02.631108 26069 ts_tablet_manager.cc:1397] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:02.631044 26068 raft_consensus.cc:348] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.631160 26068 raft_consensus.cc:374] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:02.631206 26068 raft_consensus.cc:729] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c925bec3634962983bc3c15a783d40, State: Initialized, Role: FOLLOWER
I20260929 16:31:02.631330 26070 tablet_bootstrap.cc:654] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:02.631356 26068 consensus_queue.cc:260] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.631573 26069 raft_consensus.cc:348] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.631691 26069 raft_consensus.cc:374] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:02.631731 26069 raft_consensus.cc:729] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad69a60d0e7b478b89a501ed37549bef, State: Initialized, Role: FOLLOWER
I20260929 16:31:02.631865 26069 consensus_queue.cc:260] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.632067 26068 ts_tablet_manager.cc:1428] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:02.632215 26069 ts_tablet_manager.cc:1428] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:02.633145 26070 tablet_bootstrap.cc:492] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc: No bootstrap required, opened a new log
I20260929 16:31:02.633239 26070 ts_tablet_manager.cc:1397] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 16:31:02.633744 26070 raft_consensus.cc:348] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.633867 26070 raft_consensus.cc:374] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:02.633908 26070 raft_consensus.cc:729] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ba605335af44aab6b74b703aba6cdc, State: Initialized, Role: FOLLOWER
I20260929 16:31:02.634040 26070 consensus_queue.cc:260] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.634332 26070 ts_tablet_manager.cc:1428] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:31:02.744652 26052 raft_consensus.cc:482] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:02.744782 26052 raft_consensus.cc:504] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.745322 26052 leader_election.cc:288] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039), 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839)
I20260929 16:31:02.745802 25630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f43c4d7ba5441288dd16cb0d2291dd0" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad69a60d0e7b478b89a501ed37549bef" is_pre_election: true
I20260929 16:31:02.745857 25910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f43c4d7ba5441288dd16cb0d2291dd0" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c925bec3634962983bc3c15a783d40" is_pre_election: true
I20260929 16:31:02.745971 25630 raft_consensus.cc:2454] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 0.
I20260929 16:31:02.746048 25910 raft_consensus.cc:2454] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 0.
I20260929 16:31:02.746372 25703 leader_election.cc:302] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad69a60d0e7b478b89a501ed37549bef, c1ba605335af44aab6b74b703aba6cdc; no voters:
I20260929 16:31:02.746539 26052 raft_consensus.cc:2781] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:02.746601 26052 raft_consensus.cc:482] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:02.746651 26052 raft_consensus.cc:3037] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:02.747604 26052 raft_consensus.cc:504] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.747921 26052 leader_election.cc:288] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 election: Requested vote from peers ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039), 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839)
I20260929 16:31:02.748202 25630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f43c4d7ba5441288dd16cb0d2291dd0" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad69a60d0e7b478b89a501ed37549bef"
I20260929 16:31:02.748250 25910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f43c4d7ba5441288dd16cb0d2291dd0" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c925bec3634962983bc3c15a783d40"
I20260929 16:31:02.748346 25910 raft_consensus.cc:3037] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:02.748347 25630 raft_consensus.cc:3037] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:02.749584 25910 raft_consensus.cc:2454] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 1.
I20260929 16:31:02.749584 25630 raft_consensus.cc:2454] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 1.
I20260929 16:31:02.749889 25702 leader_election.cc:302] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 54c925bec3634962983bc3c15a783d40, ad69a60d0e7b478b89a501ed37549bef, c1ba605335af44aab6b74b703aba6cdc; no voters:
I20260929 16:31:02.750114 26052 raft_consensus.cc:2781] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:02.750224 26052 raft_consensus.cc:686] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 1 LEADER]: Becoming Leader. State: Replica: c1ba605335af44aab6b74b703aba6cdc, State: Running, Role: LEADER
I20260929 16:31:02.750368 26052 consensus_queue.cc:237] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.751864 25474 catalog_manager.cc:5495] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc reported cstate change: term changed from 0 to 1, leader changed from <none> to c1ba605335af44aab6b74b703aba6cdc (127.23.44.2). New cstate: current_term: 1 leader_uuid: "c1ba605335af44aab6b74b703aba6cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20260929 16:31:02.790059 26078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:02.786706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.23.44.2:43937)
0929 16:31:02.786968 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:02.786975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:02.786987 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:31:02.787024 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:02.787026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:02.787069 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:02.787142 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:02.787388 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:02.787914 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:02.788922 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:02.789175 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:02.789349 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:02.789568 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:02.789631 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:02.789689 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:02.789801 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 16:31:02.789844 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":125,"threads_started":1}
I20260929 16:31:02.796010 25630 raft_consensus.cc:1261] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Refusing update from remote peer c1ba605335af44aab6b74b703aba6cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:02.796010 25910 raft_consensus.cc:1261] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Refusing update from remote peer c1ba605335af44aab6b74b703aba6cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:02.796384 26052 consensus_queue.cc:1035] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:02.796533 26048 consensus_queue.cc:1035] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:02.825947 25472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:02.826099 25472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:02.827975 25472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b: Sending DeleteTablet for 3 replicas of tablet 8f43c4d7ba5441288dd16cb0d2291dd0
I20260929 16:31:02.828418 25611 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f43c4d7ba5441288dd16cb0d2291dd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:02 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:33088
I20260929 16:31:02.828495 25751 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f43c4d7ba5441288dd16cb0d2291dd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:02 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:44390
I20260929 16:31:02.828588 26057 tablet_replica.cc:330] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef: stopping tablet replica
I20260929 16:31:02.828573 25888 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f43c4d7ba5441288dd16cb0d2291dd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:02 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:60804
I20260929 16:31:02.828693 26057 raft_consensus.cc:2229] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:02.828715 26059 tablet_replica.cc:330] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40: stopping tablet replica
I20260929 16:31:02.828720 26058 tablet_replica.cc:330] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc: stopping tablet replica
I20260929 16:31:02.828820 26057 raft_consensus.cc:2258] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:02.828825 26059 raft_consensus.cc:2229] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:02.828821 26058 raft_consensus.cc:2229] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:02.828912 26059 raft_consensus.cc:2258] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:02.828976 26058 raft_consensus.cc:2258] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:02.829540 26058 ts_tablet_manager.cc:1905] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:02.829691 26059 ts_tablet_manager.cc:1905] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:02.829792 26057 ts_tablet_manager.cc:1905] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:02.831882 26058 ts_tablet_manager.cc:1918] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:31:02.831943 26058 log.cc:1196] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/8f43c4d7ba5441288dd16cb0d2291dd0
I20260929 16:31:02.832039 26059 ts_tablet_manager.cc:1918] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:31:02.832034 26057 ts_tablet_manager.cc:1918] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:31:02.832103 26059 log.cc:1196] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/8f43c4d7ba5441288dd16cb0d2291dd0
I20260929 16:31:02.832103 26057 log.cc:1196] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-0/wal/wals/8f43c4d7ba5441288dd16cb0d2291dd0
I20260929 16:31:02.832186 26058 ts_tablet_manager.cc:1939] T 8f43c4d7ba5441288dd16cb0d2291dd0 P c1ba605335af44aab6b74b703aba6cdc: Deleting consensus metadata
I20260929 16:31:02.832360 26059 ts_tablet_manager.cc:1939] T 8f43c4d7ba5441288dd16cb0d2291dd0 P 54c925bec3634962983bc3c15a783d40: Deleting consensus metadata
I20260929 16:31:02.832360 26057 ts_tablet_manager.cc:1939] T 8f43c4d7ba5441288dd16cb0d2291dd0 P ad69a60d0e7b478b89a501ed37549bef: Deleting consensus metadata
I20260929 16:31:02.832726 25461 catalog_manager.cc:4841] TS c1ba605335af44aab6b74b703aba6cdc (127.23.44.2:43937): tablet 8f43c4d7ba5441288dd16cb0d2291dd0 (table test-table [id=4f6186ca5efb48ffbe94f59a76471757]) successfully deleted
I20260929 16:31:02.832935 25458 catalog_manager.cc:4841] TS 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839): tablet 8f43c4d7ba5441288dd16cb0d2291dd0 (table test-table [id=4f6186ca5efb48ffbe94f59a76471757]) successfully deleted
I20260929 16:31:02.833223 25459 catalog_manager.cc:4841] TS ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039): tablet 8f43c4d7ba5441288dd16cb0d2291dd0 (table test-table [id=4f6186ca5efb48ffbe94f59a76471757]) successfully deleted
I20260929 16:31:02.833932 25472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:02.834231 25472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:02.838016 25611 tablet_service.cc:1467] Processing CreateTablet for tablet 02efaa3ceec946499e654b0df235f008 (DEFAULT_TABLE table=test-table [id=ec08c0ccd4a84d1386cc238211331ac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:02.838169 25751 tablet_service.cc:1467] Processing CreateTablet for tablet 02efaa3ceec946499e654b0df235f008 (DEFAULT_TABLE table=test-table [id=ec08c0ccd4a84d1386cc238211331ac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:02.838254 25611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02efaa3ceec946499e654b0df235f008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:02.838259 25888 tablet_service.cc:1467] Processing CreateTablet for tablet 02efaa3ceec946499e654b0df235f008 (DEFAULT_TABLE table=test-table [id=ec08c0ccd4a84d1386cc238211331ac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:02.838368 25751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02efaa3ceec946499e654b0df235f008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:02.838434 25888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02efaa3ceec946499e654b0df235f008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:02.840005 26068 tablet_bootstrap.cc:492] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40: Bootstrap starting.
I20260929 16:31:02.840366 26070 tablet_bootstrap.cc:492] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc: Bootstrap starting.
I20260929 16:31:02.840826 26068 tablet_bootstrap.cc:654] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:02.841060 26070 tablet_bootstrap.cc:654] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:02.841411 26069 tablet_bootstrap.cc:492] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef: Bootstrap starting.
I20260929 16:31:02.841867 26068 tablet_bootstrap.cc:492] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40: No bootstrap required, opened a new log
I20260929 16:31:02.841939 26068 ts_tablet_manager.cc:1397] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:31:02.842290 26069 tablet_bootstrap.cc:654] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:02.842304 26068 raft_consensus.cc:348] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.842401 26068 raft_consensus.cc:374] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:02.842437 26068 raft_consensus.cc:729] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c925bec3634962983bc3c15a783d40, State: Initialized, Role: FOLLOWER
I20260929 16:31:02.842578 26068 consensus_queue.cc:260] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.842876 26068 ts_tablet_manager.cc:1428] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:31:02.843572 26070 tablet_bootstrap.cc:492] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc: No bootstrap required, opened a new log
I20260929 16:31:02.843655 26070 ts_tablet_manager.cc:1397] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260929 16:31:02.844043 26070 raft_consensus.cc:348] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.844158 26070 raft_consensus.cc:374] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:02.844197 26070 raft_consensus.cc:729] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ba605335af44aab6b74b703aba6cdc, State: Initialized, Role: FOLLOWER
I20260929 16:31:02.844318 26070 consensus_queue.cc:260] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.844617 26070 ts_tablet_manager.cc:1428] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 16:31:02.845702 26069 tablet_bootstrap.cc:492] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef: No bootstrap required, opened a new log
I20260929 16:31:02.845784 26069 ts_tablet_manager.cc:1397] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 16:31:02.846205 26069 raft_consensus.cc:348] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.846316 26069 raft_consensus.cc:374] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:02.846357 26069 raft_consensus.cc:729] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad69a60d0e7b478b89a501ed37549bef, State: Initialized, Role: FOLLOWER
I20260929 16:31:02.846484 26069 consensus_queue.cc:260] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.846755 26069 ts_tablet_manager.cc:1428] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:02.921097 26048 raft_consensus.cc:482] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:02.921226 26048 raft_consensus.cc:504] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.921718 26048 leader_election.cc:288] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039), 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839)
I20260929 16:31:02.922140 25630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02efaa3ceec946499e654b0df235f008" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad69a60d0e7b478b89a501ed37549bef" is_pre_election: true
I20260929 16:31:02.922176 25910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02efaa3ceec946499e654b0df235f008" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c925bec3634962983bc3c15a783d40" is_pre_election: true
I20260929 16:31:02.922310 25630 raft_consensus.cc:2454] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 0.
I20260929 16:31:02.922310 25910 raft_consensus.cc:2454] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 0.
I20260929 16:31:02.922644 25703 leader_election.cc:302] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad69a60d0e7b478b89a501ed37549bef, c1ba605335af44aab6b74b703aba6cdc; no voters:
I20260929 16:31:02.922873 26048 raft_consensus.cc:2781] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:02.922923 26048 raft_consensus.cc:482] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:02.922969 26048 raft_consensus.cc:3037] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:02.923988 26048 raft_consensus.cc:504] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.924306 26048 leader_election.cc:288] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 election: Requested vote from peers ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039), 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839)
I20260929 16:31:02.924657 25910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02efaa3ceec946499e654b0df235f008" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c925bec3634962983bc3c15a783d40"
I20260929 16:31:02.924657 25630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02efaa3ceec946499e654b0df235f008" candidate_uuid: "c1ba605335af44aab6b74b703aba6cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad69a60d0e7b478b89a501ed37549bef"
I20260929 16:31:02.924778 25910 raft_consensus.cc:3037] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:02.924778 25630 raft_consensus.cc:3037] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:02.925972 25630 raft_consensus.cc:2454] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 1.
I20260929 16:31:02.925982 25910 raft_consensus.cc:2454] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1ba605335af44aab6b74b703aba6cdc in term 1.
I20260929 16:31:02.926304 25703 leader_election.cc:302] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 54c925bec3634962983bc3c15a783d40, ad69a60d0e7b478b89a501ed37549bef, c1ba605335af44aab6b74b703aba6cdc; no voters:
I20260929 16:31:02.926517 26048 raft_consensus.cc:2781] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:02.926607 26048 raft_consensus.cc:686] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 1 LEADER]: Becoming Leader. State: Replica: c1ba605335af44aab6b74b703aba6cdc, State: Running, Role: LEADER
I20260929 16:31:02.926774 26048 consensus_queue.cc:237] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } }
I20260929 16:31:02.928512 25472 catalog_manager.cc:5495] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc reported cstate change: term changed from 0 to 1, leader changed from <none> to c1ba605335af44aab6b74b703aba6cdc (127.23.44.2). New cstate: current_term: 1 leader_uuid: "c1ba605335af44aab6b74b703aba6cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1ba605335af44aab6b74b703aba6cdc" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20260929 16:31:02.950340 25910 raft_consensus.cc:1261] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Refusing update from remote peer c1ba605335af44aab6b74b703aba6cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:02.950456 25631 raft_consensus.cc:1261] T 02efaa3ceec946499e654b0df235f008 P ad69a60d0e7b478b89a501ed37549bef [term 1 FOLLOWER]: Refusing update from remote peer c1ba605335af44aab6b74b703aba6cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:02.950781 26052 consensus_queue.cc:1035] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "54c925bec3634962983bc3c15a783d40" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:02.951090 26052 consensus_queue.cc:1035] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad69a60d0e7b478b89a501ed37549bef" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:31:02.962849 26040 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 16:31:04.511260 25474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:04.511401 25474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:04.513779 25474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fa3d5830fb0436fb5b7af02bead0e3b: Sending DeleteTablet for 3 replicas of tablet 02efaa3ceec946499e654b0df235f008
I20260929 16:31:04.514416 25751 tablet_service.cc:1514] Processing DeleteTablet for tablet 02efaa3ceec946499e654b0df235f008 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:04 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:44390
I20260929 16:31:04.514430 25611 tablet_service.cc:1514] Processing DeleteTablet for tablet 02efaa3ceec946499e654b0df235f008 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:04 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:33088
I20260929 16:31:04.514494 25888 tablet_service.cc:1514] Processing DeleteTablet for tablet 02efaa3ceec946499e654b0df235f008 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:04 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:60804
I20260929 16:31:04.514827 26111 tablet_replica.cc:330] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40: stopping tablet replica
I20260929 16:31:04.515049 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 25546
I20260929 16:31:04.516311 26111 raft_consensus.cc:2229] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:04.516474 26111 raft_consensus.cc:2258] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:04.517143 26111 ts_tablet_manager.cc:1905] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:04.519747 26111 ts_tablet_manager.cc:1918] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:31:04.519845 26111 log.cc:1196] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/02efaa3ceec946499e654b0df235f008
I20260929 16:31:04.520134 26111 ts_tablet_manager.cc:1939] T 02efaa3ceec946499e654b0df235f008 P 54c925bec3634962983bc3c15a783d40: Deleting consensus metadata
I20260929 16:31:04.520841 25458 catalog_manager.cc:4841] TS 54c925bec3634962983bc3c15a783d40 (127.23.44.3:39839): tablet 02efaa3ceec946499e654b0df235f008 (table test-table [id=ec08c0ccd4a84d1386cc238211331ac4]) successfully deleted
I20260929 16:31:04.521509 26109 tablet_replica.cc:330] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc: stopping tablet replica
I20260929 16:31:04.521661 26109 raft_consensus.cc:2229] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:04.521854 26109 raft_consensus.cc:2258] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:04.522468 26109 ts_tablet_manager.cc:1905] T 02efaa3ceec946499e654b0df235f008 P c1ba605335af44aab6b74b703aba6cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 16:31:04.522763 25459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:46039 (error 108)
I20260929 16:31:04.522814 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 25686
I20260929 16:31:04.524490 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.524019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:46039 (local address 127.0.0.1:33092)
0929 16:31:04.524273 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:04.524361 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":77,"threads_started":1}
W20260929 16:31:04.524641 25459 catalog_manager.cc:4584] TS ad69a60d0e7b478b89a501ed37549bef (127.23.44.1:46039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02efaa3ceec946499e654b0df235f008: Network error: Client connection negotiation failed: client connection to 127.23.44.1:46039: connect: Connection refused (error 111)
I20260929 16:31:04.530581 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 25826
I20260929 16:31:04.531352 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.530853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43937 (local address 127.0.0.1:44412)
0929 16:31:04.531029 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:04.531253 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.2:43937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 16:31:04.537484 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 25442
2026-09-29T16:31:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6340 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:04 dist-test-slave-mlmz krb5kdc[26116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:04 dist-test-slave-mlmz krb5kdc[26116](info): set up 2 sockets
Sep 29 16:31:04 dist-test-slave-mlmz krb5kdc[26116](info): commencing operation
krb5kdc: starting...
W20260929 16:31:05.581085 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:31:05 dist-test-slave-mlmz krb5kdc[26116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:31:05.701367 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:45965
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:45965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:05.812142 26129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:05.812404 26129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:05.812449 26129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:05.816504 26129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:05.816601 26129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:05.816627 26129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:05.816643 26129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:05.816659 26129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:05.821841 26129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:45965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:45965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:05.822973 26129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:05.823956 26129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:31:05.833567 26129 server_base.cc:1034] running on GCE node
W20260929 16:31:05.833540 26135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:05.833760 26137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:05.833760 26134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:05.834435 26129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:05.835537 26129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:05.836758 26129 hybrid_clock.cc:648] HybridClock initialized: now 1790699465836726 us; error 45 us; skew 500 ppm
Sep 29 16:31:05 dist-test-slave-mlmz krb5kdc[26116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699465, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:05.839635 26129 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:31:05.840574 26129 webserver.cc:460] Webserver started at http://127.23.44.62:33121/ using document root <none> and password file <none>
I20260929 16:31:05.841146 26129 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:05.841241 26129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:05.841480 26129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:05.843478 26129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "bfb3c448a8ab416bb6fa006d20e95219"
format_stamp: "Formatted at 2026-09-29 16:31:05 on dist-test-slave-mlmz"
server_key: "0359bf8e623c36e9f612e6ffc197f28b"
server_key_iv: "a6ec57bcf3abf7f534fa5c39f43914d8"
server_key_version: "encryptionkey@0"
I20260929 16:31:05.844046 26129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "bfb3c448a8ab416bb6fa006d20e95219"
format_stamp: "Formatted at 2026-09-29 16:31:05 on dist-test-slave-mlmz"
server_key: "0359bf8e623c36e9f612e6ffc197f28b"
server_key_iv: "a6ec57bcf3abf7f534fa5c39f43914d8"
server_key_version: "encryptionkey@0"
I20260929 16:31:05.847805 26129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 16:31:05.850464 26144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:05.851707 26129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:31:05.851856 26129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "bfb3c448a8ab416bb6fa006d20e95219"
format_stamp: "Formatted at 2026-09-29 16:31:05 on dist-test-slave-mlmz"
server_key: "0359bf8e623c36e9f612e6ffc197f28b"
server_key_iv: "a6ec57bcf3abf7f534fa5c39f43914d8"
server_key_version: "encryptionkey@0"
I20260929 16:31:05.851980 26129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:05.870548 26129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:05.873878 26129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:05.874087 26129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:05.881053 26129 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:45965
I20260929 16:31:05.881079 26196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:45965 every 8 connection(s)
I20260929 16:31:05.881989 26129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:05.884609 26197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:05.887480 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 26129
I20260929 16:31:05.887614 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:05.887866 23728 external_mini_cluster.cc:1428] Setting key 297395a448161cc3dc38ccd5ebbdd8a1
I20260929 16:31:05.890657 26197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219: Bootstrap starting.
I20260929 16:31:05.892740 26197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:05.893673 26197 log.cc:824] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:05.895835 26197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219: No bootstrap required, opened a new log
Sep 29 16:31:05 dist-test-slave-mlmz krb5kdc[26116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:05.899312 26197 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb3c448a8ab416bb6fa006d20e95219" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45965 } }
I20260929 16:31:05.899513 26197 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:05.899600 26197 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb3c448a8ab416bb6fa006d20e95219, State: Initialized, Role: FOLLOWER
I20260929 16:31:05.900030 26197 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb3c448a8ab416bb6fa006d20e95219" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45965 } }
I20260929 16:31:05.900172 26197 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:05.900238 26197 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:05.900345 26197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:05.901510 26197 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb3c448a8ab416bb6fa006d20e95219" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45965 } }
I20260929 16:31:05.901866 26197 leader_election.cc:302] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfb3c448a8ab416bb6fa006d20e95219; no voters:
I20260929 16:31:05.902184 26197 leader_election.cc:288] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:05.902279 26202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:05.902516 26202 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [term 1 LEADER]: Becoming Leader. State: Replica: bfb3c448a8ab416bb6fa006d20e95219, State: Running, Role: LEADER
I20260929 16:31:05.902580 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:05.889238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.23.44.62:45965)
0929 16:31:05.889683 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:05.889697 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:05.889730 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:31:05.890318 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:05.890334 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:05.890664 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:05.891033 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:05.891813 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:05.892788 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:05.894069 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:05.894681 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:05.894802 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:05.897552 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:05.897557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:05.897569 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:05.897595 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:05.899325 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:05.899844 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:05.899852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:05.899859 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:05.899992 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:05.900291 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:05.900295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:05.900296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:05.900807 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:05.901007 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:05.901298 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 16:31:05.901604 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
I20260929 16:31:05.902837 26202 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb3c448a8ab416bb6fa006d20e95219" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45965 } }
I20260929 16:31:05.903393 26197 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:05.904405 26203 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfb3c448a8ab416bb6fa006d20e95219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb3c448a8ab416bb6fa006d20e95219" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45965 } } }
I20260929 16:31:05.904520 26203 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:05.904902 26204 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfb3c448a8ab416bb6fa006d20e95219. Latest consensus state: current_term: 1 leader_uuid: "bfb3c448a8ab416bb6fa006d20e95219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb3c448a8ab416bb6fa006d20e95219" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45965 } } }
I20260929 16:31:05.905046 26204 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219 [sys.catalog]: This master's current role is: LEADER
W20260929 16:31:05.907971 26217 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:31:05.908061 26217 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:31:05.908138 26211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:05.908819 26211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:05.914142 26211 catalog_manager.cc:1347] Generated new cluster ID: 3b6d7fc91e7e43e29f76261da8fa3de4
I20260929 16:31:05.914206 26211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:05.948338 26211 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:05.949749 26211 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:05.955814 26211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfb3c448a8ab416bb6fa006d20e95219: Generated new TSK 0
I20260929 16:31:05.956808 26211 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:31:05.997493 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:45965
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:06.106298 26223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:06.106560 26223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:06.106634 26223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:06.110611 26223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:06.110692 26223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:06.110778 26223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:06.115803 26223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:45965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:06.116789 26223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:06.117818 26223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:06.127095 26228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:06.127209 26223 server_base.cc:1034] running on GCE node
W20260929 16:31:06.127113 26231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:06.127099 26229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:06.127737 26223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:06.128386 26223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:06.129571 26223 hybrid_clock.cc:648] HybridClock initialized: now 1790699466129557 us; error 29 us; skew 500 ppm
Sep 29 16:31:06 dist-test-slave-mlmz krb5kdc[26116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699466, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:06.132315 26223 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:31:06.133196 26223 webserver.cc:460] Webserver started at http://127.23.44.1:38107/ using document root <none> and password file <none>
I20260929 16:31:06.133750 26223 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:06.133838 26223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:06.134064 26223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:06.136001 26223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "106a53787a4242dbbb7f3f10d9bc66fa"
format_stamp: "Formatted at 2026-09-29 16:31:06 on dist-test-slave-mlmz"
server_key: "7f7f82cd9a9afb80fbd454ccb7380311"
server_key_iv: "8f5945b12d421c3445cedfaa449ddb8c"
server_key_version: "encryptionkey@0"
I20260929 16:31:06.136566 26223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "106a53787a4242dbbb7f3f10d9bc66fa"
format_stamp: "Formatted at 2026-09-29 16:31:06 on dist-test-slave-mlmz"
server_key: "7f7f82cd9a9afb80fbd454ccb7380311"
server_key_iv: "8f5945b12d421c3445cedfaa449ddb8c"
server_key_version: "encryptionkey@0"
I20260929 16:31:06.140133 26223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:31:06.142513 26238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:06.143739 26223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:31:06.143870 26223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "106a53787a4242dbbb7f3f10d9bc66fa"
format_stamp: "Formatted at 2026-09-29 16:31:06 on dist-test-slave-mlmz"
server_key: "7f7f82cd9a9afb80fbd454ccb7380311"
server_key_iv: "8f5945b12d421c3445cedfaa449ddb8c"
server_key_version: "encryptionkey@0"
I20260929 16:31:06.143973 26223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:06.165807 26223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:06.168781 26223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:06.168987 26223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:06.169603 26223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:06.170392 26223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:06.170463 26223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:06.170531 26223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:06.170574 26223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:06.179076 26223 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:44095
I20260929 16:31:06.179101 26351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:44095 every 8 connection(s)
I20260929 16:31:06.179893 26223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:06.183626 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 26223
I20260929 16:31:06.183723 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:06.184345 23728 external_mini_cluster.cc:1428] Setting key 5555a8e7b0b0d1aad1fe7ee69d12293b
Sep 29 16:31:06 dist-test-slave-mlmz krb5kdc[26116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699466, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
I20260929 16:31:06.194061 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:06.181658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:59499 (local address 127.23.44.62:45965)
0929 16:31:06.181806 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:06.181810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:06.182791 (+ 981us) server_negotiation.cc:407] Connection header received
0929 16:31:06.183915 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:06.183919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:06.183975 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:06.184067 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:06.185343 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:06.186007 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:06.187172 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:06.187523 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:06.187609 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:06.190273 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:06.190277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:06.190280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:06.190303 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:06.191906 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:06.192470 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:06.192473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:06.192475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:06.192564 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:06.192931 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:06.192936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:06.192937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:06.193130 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:06.193236 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:06.193769 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 16:31:06.193888 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:31:06.194855 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:06.181993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:45965 (local address 127.23.44.1:59499)
0929 16:31:06.182645 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:06.182685 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:06.183463 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:06.184229 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:06.184236 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:06.184577 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:06.185172 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:06.185191 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:06.186161 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:06.186164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:06.187036 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:06.187045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:06.187682 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:06.187688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:06.187940 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:06.188581 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:06.188606 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:06.190109 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:06.192049 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:06.192055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:06.192069 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:06.192357 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:06.192685 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:06.192688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:06.192691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:06.192832 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:06.193247 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:06.193255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:06.193504 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 16:31:06.193698 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 16:31:06.194006 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":106,"threads_started":1}
I20260929 16:31:06.195961 26352 heartbeater.cc:347] Connected to a master server at 127.23.44.62:45965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
I20260929 16:31:06.196216 26352 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:06.196669 26352 heartbeater.cc:511] Master 127.23.44.62:45965 requested a full tablet report, sending...
I20260929 16:31:06.198320 26161 ts_manager.cc:194] Registered new tserver with Master: 106a53787a4242dbbb7f3f10d9bc66fa (127.23.44.1:44095)
I20260929 16:31:06.199491 26161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:59499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:31:06.212265 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:45965
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:06.320371 26357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:06.320600 26357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:06.320642 26357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:06.324496 26357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:06.324542 26357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:06.324595 26357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:06.329645 26357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:45965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:06.330529 26357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:06.331419 26357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:31:06.341615 26357 server_base.cc:1034] running on GCE node
W20260929 16:31:06.341667 26365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:06.341598 26363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:06.341620 26362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:06.342201 26357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:06.342790 26357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:06.343969 26357 hybrid_clock.cc:648] HybridClock initialized: now 1790699466343938 us; error 42 us; skew 500 ppm
Sep 29 16:31:06 dist-test-slave-mlmz krb5kdc[26116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699466, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:06.347008 26357 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:31:06.348369 26357 webserver.cc:460] Webserver started at http://127.23.44.2:42181/ using document root <none> and password file <none>
I20260929 16:31:06.348927 26357 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:06.348989 26357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:06.349218 26357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:06.351449 26357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "c96906dc3a2f4623b870fe1e096cc954"
format_stamp: "Formatted at 2026-09-29 16:31:06 on dist-test-slave-mlmz"
server_key: "20c75d0cbbcf3b2625be9c9fb8640889"
server_key_iv: "1ecca589f1b4dabd29ab8292ef7cb06b"
server_key_version: "encryptionkey@0"
I20260929 16:31:06.352049 26357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "c96906dc3a2f4623b870fe1e096cc954"
format_stamp: "Formatted at 2026-09-29 16:31:06 on dist-test-slave-mlmz"
server_key: "20c75d0cbbcf3b2625be9c9fb8640889"
server_key_iv: "1ecca589f1b4dabd29ab8292ef7cb06b"
server_key_version: "encryptionkey@0"
I20260929 16:31:06.355690 26357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:31:06.358198 26372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:06.359404 26357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:06.359529 26357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "c96906dc3a2f4623b870fe1e096cc954"
format_stamp: "Formatted at 2026-09-29 16:31:06 on dist-test-slave-mlmz"
server_key: "20c75d0cbbcf3b2625be9c9fb8640889"
server_key_iv: "1ecca589f1b4dabd29ab8292ef7cb06b"
server_key_version: "encryptionkey@0"
I20260929 16:31:06.359632 26357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:06.379432 26357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:06.382643 26357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:06.382836 26357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:06.383389 26357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:06.384176 26357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:06.384222 26357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:06.384259 26357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:06.384316 26357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:06.392669 26357 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:40385
I20260929 16:31:06.392692 26485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:40385 every 8 connection(s)
I20260929 16:31:06.393550 26357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:06.398092 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 26357
I20260929 16:31:06.398245 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:06.398577 23728 external_mini_cluster.cc:1428] Setting key 0aed772691e5110c0f94b6b5924e22a3
Sep 29 16:31:06 dist-test-slave-mlmz krb5kdc[26116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699466, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:31:06.408190 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:06.395217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:43687 (local address 127.23.44.62:45965)
0929 16:31:06.395363 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:06.395368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:06.396384 (+ 1016us) server_negotiation.cc:407] Connection header received
0929 16:31:06.397220 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:06.397223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:06.397328 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:06.397415 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:06.398601 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:06.399277 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:06.400655 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:06.400900 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:06.401008 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:06.404175 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:06.404179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:06.404182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:06.404212 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:06.405946 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:06.406588 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:06.406590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:06.406591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:06.406661 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:06.407044 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:06.407047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:06.407048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:06.407210 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:06.407282 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:06.407911 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 16:31:06.408004 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:31:06.409006 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:06.395613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:45965 (local address 127.23.44.2:43687)
0929 16:31:06.396243 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:06.396282 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:06.397015 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:06.397554 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:06.397560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:06.397914 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:06.398453 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:06.398468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:06.399415 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:06.399418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:06.400530 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:06.400540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:06.401074 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:06.401080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:06.401229 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:06.401886 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:06.401909 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:06.403996 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:06.406061 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:06.406069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:06.406085 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:06.406467 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:06.406751 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:06.406754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:06.406756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:06.406948 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:06.407318 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:06.407324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:06.407659 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 16:31:06.407869 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 16:31:06.408203 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":106,"threads_started":1}
I20260929 16:31:06.410089 26486 heartbeater.cc:347] Connected to a master server at 127.23.44.62:45965
I20260929 16:31:06.410328 26486 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
I20260929 16:31:06.410837 26486 heartbeater.cc:511] Master 127.23.44.62:45965 requested a full tablet report, sending...
I20260929 16:31:06.411948 26161 ts_manager.cc:194] Registered new tserver with Master: c96906dc3a2f4623b870fe1e096cc954 (127.23.44.2:40385)
I20260929 16:31:06.412585 26161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:43687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:31:06.426218 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:45965
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:06.533164 26491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:06.533452 26491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:06.533496 26491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:06.537590 26491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:06.537637 26491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:06.537693 26491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:06.543147 26491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:45965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:06.544003 26491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:06.544859 26491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:06.555984 26496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:06.556326 26499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:06.556561 26497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:06.556998 26491 server_base.cc:1034] running on GCE node
I20260929 16:31:06.557456 26491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:06.558035 26491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:06.559249 26491 hybrid_clock.cc:648] HybridClock initialized: now 1790699466559244 us; error 33 us; skew 500 ppm
Sep 29 16:31:06 dist-test-slave-mlmz krb5kdc[26116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699466, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:06.562151 26491 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:31:06.563009 26491 webserver.cc:460] Webserver started at http://127.23.44.3:38721/ using document root <none> and password file <none>
I20260929 16:31:06.563603 26491 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:06.563668 26491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:06.563834 26491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:06.565812 26491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "53f2985e515848b785cf717473376755"
format_stamp: "Formatted at 2026-09-29 16:31:06 on dist-test-slave-mlmz"
server_key: "063ff159088401a5384ced93a992fbb2"
server_key_iv: "2bd67edaa916cf80c19e25efddfd3505"
server_key_version: "encryptionkey@0"
I20260929 16:31:06.566326 26491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "53f2985e515848b785cf717473376755"
format_stamp: "Formatted at 2026-09-29 16:31:06 on dist-test-slave-mlmz"
server_key: "063ff159088401a5384ced93a992fbb2"
server_key_iv: "2bd67edaa916cf80c19e25efddfd3505"
server_key_version: "encryptionkey@0"
I20260929 16:31:06.569811 26491 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:31:06.572160 26506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:06.573374 26491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:06.573472 26491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "53f2985e515848b785cf717473376755"
format_stamp: "Formatted at 2026-09-29 16:31:06 on dist-test-slave-mlmz"
server_key: "063ff159088401a5384ced93a992fbb2"
server_key_iv: "2bd67edaa916cf80c19e25efddfd3505"
server_key_version: "encryptionkey@0"
I20260929 16:31:06.573553 26491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:06.592410 26491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:06.595296 26491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:06.595439 26491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:06.595922 26491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:06.596674 26491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:06.596719 26491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:06.596756 26491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:06.596772 26491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:06.606132 26491 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:41949
I20260929 16:31:06.606163 26619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:41949 every 8 connection(s)
I20260929 16:31:06.606971 26491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:06.611922 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 26491
I20260929 16:31:06.612046 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:06.612309 23728 external_mini_cluster.cc:1428] Setting key 2c15db7322ae2b8f1266c7b983b8d198
Sep 29 16:31:06 dist-test-slave-mlmz krb5kdc[26116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699466, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:06.619923 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:06.608633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:38089 (local address 127.23.44.62:45965)
0929 16:31:06.608768 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:06.608772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:06.609742 (+ 970us) server_negotiation.cc:407] Connection header received
0929 16:31:06.610623 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:06.610627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:06.610683 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:06.610790 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:06.611873 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:06.612555 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:06.614019 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:06.614243 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:06.614349 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:06.616737 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:06.616740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:06.616742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:06.616764 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:06.618051 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:06.618540 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:06.618543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:06.618544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:06.618626 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:06.619004 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:06.619007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:06.619008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:06.619174 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:06.619253 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:06.619721 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 16:31:06.619812 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:31:06.620653 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:06.608913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:45965 (local address 127.23.44.3:38089)
0929 16:31:06.609598 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:06.609638 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:06.610423 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:06.610947 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:06.610960 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:06.611286 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:06.611725 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:06.611737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:06.612674 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:06.612677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:06.613901 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:06.613910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:06.614382 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:06.614386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:06.614558 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:06.615144 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:06.615159 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:06.616590 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:06.618165 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:06.618170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:06.618180 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:06.618433 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:06.618758 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:06.618760 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:06.618762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:06.618894 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:06.619267 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:06.619277 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:06.619504 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 16:31:06.619663 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 16:31:06.619887 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":116,"threads_started":1}
I20260929 16:31:06.621804 26620 heartbeater.cc:347] Connected to a master server at 127.23.44.62:45965
I20260929 16:31:06.622041 26620 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:06.622478 26620 heartbeater.cc:511] Master 127.23.44.62:45965 requested a full tablet report, sending...
I20260929 16:31:06.623482 26161 ts_manager.cc:194] Registered new tserver with Master: 53f2985e515848b785cf717473376755 (127.23.44.3:41949)
I20260929 16:31:06.624146 26161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:38089
I20260929 16:31:06.626349 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:31:06.639921 26200 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:06.638042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.23.44.62:45965)
0929 16:31:06.638179 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:06.638183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:06.638394 (+ 211us) server_negotiation.cc:407] Connection header received
0929 16:31:06.638470 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:06.638472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:06.638512 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:06.638699 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:06.639766 (+ 1067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57822: BlockingRecv error: recv got EOF from 127.0.0.1:57822 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:31:06.641179 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 26223
I20260929 16:31:06.646790 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 26357
I20260929 16:31:06.651907 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 26491
I20260929 16:31:06.657038 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 26129
2026-09-29T16:31:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2114 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:06 dist-test-slave-mlmz krb5kdc[26632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:06 dist-test-slave-mlmz krb5kdc[26632](info): set up 2 sockets
Sep 29 16:31:06 dist-test-slave-mlmz krb5kdc[26632](info): commencing operation
krb5kdc: starting...
W20260929 16:31:07.701114 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:31:07 dist-test-slave-mlmz krb5kdc[26632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:31:07.820937 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39075
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:07.927835 26645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:07.928079 26645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:07.928118 26645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:07.932085 26645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:07.932138 26645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:07.932156 26645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:07.932173 26645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:07.932188 26645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:07.937191 26645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:39075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:07.938247 26645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:07.939209 26645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:07.947587 26653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:07.947587 26650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:07.947901 26645 server_base.cc:1034] running on GCE node
W20260929 16:31:07.947654 26651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:07.948681 26645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:07.949735 26645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:07.950887 26645 hybrid_clock.cc:648] HybridClock initialized: now 1790699467950860 us; error 41 us; skew 500 ppm
Sep 29 16:31:07 dist-test-slave-mlmz krb5kdc[26632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699467, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:07.962764 26645 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:31:07.963670 26645 webserver.cc:460] Webserver started at http://127.23.44.62:36277/ using document root <none> and password file <none>
I20260929 16:31:07.964228 26645 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:07.964319 26645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:07.964552 26645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:07.966706 26645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "a4b49e1807a741899c8d28d7c6971135"
format_stamp: "Formatted at 2026-09-29 16:31:07 on dist-test-slave-mlmz"
server_key: "55b8fffdfadf6f9b0221639f093de37b"
server_key_iv: "ee351537d90d090c11473dbc6df41f3a"
server_key_version: "encryptionkey@0"
I20260929 16:31:07.967284 26645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "a4b49e1807a741899c8d28d7c6971135"
format_stamp: "Formatted at 2026-09-29 16:31:07 on dist-test-slave-mlmz"
server_key: "55b8fffdfadf6f9b0221639f093de37b"
server_key_iv: "ee351537d90d090c11473dbc6df41f3a"
server_key_version: "encryptionkey@0"
I20260929 16:31:07.970827 26645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:31:07.973066 26660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:07.974268 26645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:31:07.974411 26645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "a4b49e1807a741899c8d28d7c6971135"
format_stamp: "Formatted at 2026-09-29 16:31:07 on dist-test-slave-mlmz"
server_key: "55b8fffdfadf6f9b0221639f093de37b"
server_key_iv: "ee351537d90d090c11473dbc6df41f3a"
server_key_version: "encryptionkey@0"
I20260929 16:31:07.974516 26645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:07.986001 26645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:07.989111 26645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:07.989382 26645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:07.996338 26645 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:39075
I20260929 16:31:07.996627 26712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:39075 every 8 connection(s)
I20260929 16:31:07.997339 26645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:08.000063 26713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:08.005345 26713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135: Bootstrap starting.
I20260929 16:31:08.006822 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 26645
I20260929 16:31:08.006937 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:08.007227 23728 external_mini_cluster.cc:1428] Setting key 7f92d5d7d0f545b1280b49b52317c951
I20260929 16:31:08.007918 26713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:08.008675 26713 log.cc:824] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:08.010685 26713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135: No bootstrap required, opened a new log
I20260929 16:31:08.014784 26713 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b49e1807a741899c8d28d7c6971135" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39075 } }
I20260929 16:31:08.015074 26713 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:08.015149 26713 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b49e1807a741899c8d28d7c6971135, State: Initialized, Role: FOLLOWER
I20260929 16:31:08.015650 26713 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b49e1807a741899c8d28d7c6971135" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39075 } }
I20260929 16:31:08.015792 26713 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:08.015877 26713 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:08.015974 26713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 16:31:08 dist-test-slave-mlmz krb5kdc[26632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:08.016901 26713 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b49e1807a741899c8d28d7c6971135" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39075 } }
I20260929 16:31:08.017234 26713 leader_election.cc:302] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4b49e1807a741899c8d28d7c6971135; no voters:
I20260929 16:31:08.017611 26713 leader_election.cc:288] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:08.017729 26718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:08.017932 26718 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [term 1 LEADER]: Becoming Leader. State: Replica: a4b49e1807a741899c8d28d7c6971135, State: Running, Role: LEADER
I20260929 16:31:08.018251 26718 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b49e1807a741899c8d28d7c6971135" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39075 } }
I20260929 16:31:08.018961 26713 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:08.019766 26719 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4b49e1807a741899c8d28d7c6971135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b49e1807a741899c8d28d7c6971135" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39075 } } }
I20260929 16:31:08.019896 26719 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:08.020220 26720 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b49e1807a741899c8d28d7c6971135. Latest consensus state: current_term: 1 leader_uuid: "a4b49e1807a741899c8d28d7c6971135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b49e1807a741899c8d28d7c6971135" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39075 } } }
I20260929 16:31:08.020315 26720 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:08.021238 26727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:08.022789 26716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:08.008343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.23.44.62:39075)
0929 16:31:08.008816 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:08.008834 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:08.008866 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:31:08.009661 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:08.009679 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:08.009991 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:08.010255 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:08.011241 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:08.012192 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:08.013410 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:08.013886 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:08.014022 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:08.016922 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:08.016928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:08.016940 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:08.016993 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:08.019212 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:08.019835 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:08.019840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:08.019846 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:08.019986 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:08.020321 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:08.020328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:08.020330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:08.020837 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:08.021036 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:08.021685 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 16:31:08.021981 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":156,"threads_started":1}
I20260929 16:31:08.024418 26727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:08.025543 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:31:08.029757 26727 catalog_manager.cc:1347] Generated new cluster ID: 159e90fd49c0495083a9f8eb4b2e38b4
I20260929 16:31:08.029814 26727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:08.047930 26716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:08.042222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.23.44.62:39075)
0929 16:31:08.042317 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:08.042321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:08.042595 (+ 274us) server_negotiation.cc:407] Connection header received
0929 16:31:08.042702 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:08.042705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:08.042764 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:08.042853 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:08.043378 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:08.044283 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:08.045448 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:08.045846 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:08.045951 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:08.046838 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:08.046842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:08.046845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:08.047455 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:08.047491 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:08.047512 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:08.047610 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:08.047706 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 16:31:08.047767 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 16:31:08.058286 26727 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:08.059139 26727 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:08.070680 26727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4b49e1807a741899c8d28d7c6971135: Generated new TSK 0
I20260929 16:31:08.071327 26727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:31:08.082214 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 26645
2026-09-29T16:31:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1423 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:08 dist-test-slave-mlmz krb5kdc[26746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:08 dist-test-slave-mlmz krb5kdc[26746](info): set up 2 sockets
Sep 29 16:31:08 dist-test-slave-mlmz krb5kdc[26746](info): commencing operation
krb5kdc: starting...
W20260929 16:31:09.137914 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:31:09 dist-test-slave-mlmz krb5kdc[26746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:31:09.256705 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39115
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:39115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:09.362913 26759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:09.363193 26759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:09.363273 26759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:09.367278 26759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:09.367357 26759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:09.367395 26759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:09.367455 26759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:09.367496 26759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:09.372578 26759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:39115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:09.373580 26759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:09.374549 26759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:09.382745 26764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:09.382869 26765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:09.382997 26759 server_base.cc:1034] running on GCE node
W20260929 16:31:09.382972 26767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:09.384116 26759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:09.385198 26759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:09.386436 26759 hybrid_clock.cc:648] HybridClock initialized: now 1790699469386411 us; error 42 us; skew 500 ppm
Sep 29 16:31:09 dist-test-slave-mlmz krb5kdc[26746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699469, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:09.389209 26759 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:31:09.390100 26759 webserver.cc:460] Webserver started at http://127.23.44.62:38911/ using document root <none> and password file <none>
I20260929 16:31:09.390657 26759 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:09.390748 26759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:09.390969 26759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:09.392948 26759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "cfe7b84193684502b6d5b9ce037ed2a0"
format_stamp: "Formatted at 2026-09-29 16:31:09 on dist-test-slave-mlmz"
server_key: "33087e815b420a96ac43d6e8b366fe11"
server_key_iv: "2261af2b635ef3d410c53c0c252af0f5"
server_key_version: "encryptionkey@0"
I20260929 16:31:09.393517 26759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "cfe7b84193684502b6d5b9ce037ed2a0"
format_stamp: "Formatted at 2026-09-29 16:31:09 on dist-test-slave-mlmz"
server_key: "33087e815b420a96ac43d6e8b366fe11"
server_key_iv: "2261af2b635ef3d410c53c0c252af0f5"
server_key_version: "encryptionkey@0"
I20260929 16:31:09.397233 26759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:31:09.399753 26774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:09.401368 26759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:31:09.401492 26759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "cfe7b84193684502b6d5b9ce037ed2a0"
format_stamp: "Formatted at 2026-09-29 16:31:09 on dist-test-slave-mlmz"
server_key: "33087e815b420a96ac43d6e8b366fe11"
server_key_iv: "2261af2b635ef3d410c53c0c252af0f5"
server_key_version: "encryptionkey@0"
I20260929 16:31:09.401636 26759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:09.437390 26759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:09.440462 26759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:09.440677 26759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:09.447664 26759 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:39115
I20260929 16:31:09.447945 26826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:39115 every 8 connection(s)
I20260929 16:31:09.448635 26759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:09.451314 26827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:09.452541 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 26759
I20260929 16:31:09.452711 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:09.453009 23728 external_mini_cluster.cc:1428] Setting key 192254ab716820bc8669fcc2994cd43b
I20260929 16:31:09.457154 26827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0: Bootstrap starting.
I20260929 16:31:09.459276 26827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:09.460119 26827 log.cc:824] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:31:09 dist-test-slave-mlmz krb5kdc[26746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:09.462262 26827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0: No bootstrap required, opened a new log
I20260929 16:31:09.466337 26827 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe7b84193684502b6d5b9ce037ed2a0" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39115 } }
I20260929 16:31:09.466537 26827 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:09.466616 26827 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe7b84193684502b6d5b9ce037ed2a0, State: Initialized, Role: FOLLOWER
I20260929 16:31:09.466827 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:09.454368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.23.44.62:39115)
0929 16:31:09.454786 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:09.454801 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:09.454836 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:31:09.455450 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:09.455466 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:09.455782 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:09.456118 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:09.456897 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:09.457935 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:09.458773 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:09.459237 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:09.459420 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:09.462176 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:09.462181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:09.462194 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:09.462219 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:09.463884 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:09.464440 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:09.464444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:09.464448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:09.464565 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:09.464843 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:09.464845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:09.464846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:09.465206 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:09.465441 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:09.465699 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 16:31:09.466008 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":125,"threads_started":1}
I20260929 16:31:09.467108 26827 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe7b84193684502b6d5b9ce037ed2a0" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39115 } }
I20260929 16:31:09.467253 26827 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:09.467338 26827 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:09.467432 26827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:09.468281 26827 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe7b84193684502b6d5b9ce037ed2a0" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39115 } }
I20260929 16:31:09.468611 26827 leader_election.cc:302] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfe7b84193684502b6d5b9ce037ed2a0; no voters:
I20260929 16:31:09.468896 26827 leader_election.cc:288] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:09.469179 26832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:09.469437 26832 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [term 1 LEADER]: Becoming Leader. State: Replica: cfe7b84193684502b6d5b9ce037ed2a0, State: Running, Role: LEADER
I20260929 16:31:09.469725 26832 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe7b84193684502b6d5b9ce037ed2a0" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39115 } }
I20260929 16:31:09.470053 26827 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:09.471328 26834 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfe7b84193684502b6d5b9ce037ed2a0. Latest consensus state: current_term: 1 leader_uuid: "cfe7b84193684502b6d5b9ce037ed2a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe7b84193684502b6d5b9ce037ed2a0" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39115 } } }
I20260929 16:31:09.471333 26833 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfe7b84193684502b6d5b9ce037ed2a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe7b84193684502b6d5b9ce037ed2a0" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39115 } } }
I20260929 16:31:09.471493 26834 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:09.471508 26833 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0 [sys.catalog]: This master's current role is: LEADER
W20260929 16:31:09.474344 26847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:31:09.474436 26847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:31:09.474539 26844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:09.475225 26844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:09.479815 26844 catalog_manager.cc:1347] Generated new cluster ID: ecdd0aa89d004c398f2bbd19b3dca7d9
I20260929 16:31:09.479874 26844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:09.500638 26844 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:09.501580 26844 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:09.505990 26844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfe7b84193684502b6d5b9ce037ed2a0: Generated new TSK 0
I20260929 16:31:09.506744 26844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:31:09.540024 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39115
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:09.648331 26853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:09.648573 26853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:09.648614 26853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:09.652559 26853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:09.652624 26853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:09.652680 26853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:09.657897 26853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:09.658870 26853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:09.659788 26853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:09.669232 26861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:09.669400 26859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:09.669360 26858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:09.669513 26853 server_base.cc:1034] running on GCE node
I20260929 16:31:09.669886 26853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:09.670583 26853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:09.671774 26853 hybrid_clock.cc:648] HybridClock initialized: now 1790699469671745 us; error 40 us; skew 500 ppm
Sep 29 16:31:09 dist-test-slave-mlmz krb5kdc[26746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699469, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:09.674554 26853 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:31:09.675311 26853 webserver.cc:460] Webserver started at http://127.23.44.1:35519/ using document root <none> and password file <none>
I20260929 16:31:09.675792 26853 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:09.675848 26853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:09.676011 26853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:09.677945 26853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "d1fad6b1b8904daaa1822626039ad9a9"
format_stamp: "Formatted at 2026-09-29 16:31:09 on dist-test-slave-mlmz"
server_key: "ef8d180bc3d956d2ab5957340b06117c"
server_key_iv: "d568bc189ee4a2f561540709fe674ac5"
server_key_version: "encryptionkey@0"
I20260929 16:31:09.678411 26853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "d1fad6b1b8904daaa1822626039ad9a9"
format_stamp: "Formatted at 2026-09-29 16:31:09 on dist-test-slave-mlmz"
server_key: "ef8d180bc3d956d2ab5957340b06117c"
server_key_iv: "d568bc189ee4a2f561540709fe674ac5"
server_key_version: "encryptionkey@0"
I20260929 16:31:09.681859 26853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:31:09.684202 26868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:09.685604 26853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:09.685701 26853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "d1fad6b1b8904daaa1822626039ad9a9"
format_stamp: "Formatted at 2026-09-29 16:31:09 on dist-test-slave-mlmz"
server_key: "ef8d180bc3d956d2ab5957340b06117c"
server_key_iv: "d568bc189ee4a2f561540709fe674ac5"
server_key_version: "encryptionkey@0"
I20260929 16:31:09.685783 26853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:09.702386 26853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:09.705579 26853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:09.705749 26853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:09.706261 26853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:09.707055 26853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:09.707103 26853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:09.707141 26853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:09.707156 26853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:09.716374 26853 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:43501
I20260929 16:31:09.716403 26981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:43501 every 8 connection(s)
I20260929 16:31:09.717227 26853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:09.725824 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 26853
I20260929 16:31:09.725956 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:09.726301 23728 external_mini_cluster.cc:1428] Setting key c5a73221e9f37cf881737d1e212c3b56
Sep 29 16:31:09 dist-test-slave-mlmz krb5kdc[26746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699469, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:09.731416 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:09.718983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:55977 (local address 127.23.44.62:39115)
0929 16:31:09.719138 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:09.719143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:09.720048 (+ 905us) server_negotiation.cc:407] Connection header received
0929 16:31:09.721466 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:09.721469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:09.721514 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:09.721590 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:09.722722 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:09.723281 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:09.724452 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:09.724686 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:09.724778 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:09.727563 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:09.727570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:09.727576 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:09.727607 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:09.729296 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:09.729777 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:09.729783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:09.729786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:09.729880 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:09.730197 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:09.730203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:09.730208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:09.730450 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:09.730566 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:09.731032 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 16:31:09.731168 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:31:09.731974 26984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:09.719410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39115 (local address 127.23.44.1:55977)
0929 16:31:09.719882 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:09.719929 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:09.721185 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:09.721739 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:09.721746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:09.722107 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:09.722556 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:09.722569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:09.723410 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:09.723414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:09.724300 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:09.724309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:09.724821 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:09.724823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:09.724979 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:09.725579 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:09.725601 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:09.727341 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:09.729409 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:09.729413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:09.729423 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:09.729678 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:09.729971 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:09.729973 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:09.729974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:09.730104 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:09.730569 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:09.730585 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:09.730799 (+ 214us) client_negotiation.cc:769] Sending connection context
0929 16:31:09.730949 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 16:31:09.731176 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":118,"threads_started":1}
I20260929 16:31:09.733140 26982 heartbeater.cc:347] Connected to a master server at 127.23.44.62:39115
I20260929 16:31:09.733420 26982 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:09.733985 26982 heartbeater.cc:511] Master 127.23.44.62:39115 requested a full tablet report, sending...
I20260929 16:31:09.735469 26790 ts_manager.cc:194] Registered new tserver with Master: d1fad6b1b8904daaa1822626039ad9a9 (127.23.44.1:43501)
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
I20260929 16:31:09.736603 26790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:55977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:31:09.752470 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39115
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:09.857790 26987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:09.858032 26987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:09.858073 26987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:09.861950 26987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:09.862003 26987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:09.862058 26987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:09.867085 26987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:09.867944 26987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:09.868806 26987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:09.877962 26992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:09.877987 26993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:09.878011 26987 server_base.cc:1034] running on GCE node
W20260929 16:31:09.877960 26995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:09.878633 26987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:09.879281 26987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:09.880460 26987 hybrid_clock.cc:648] HybridClock initialized: now 1790699469880445 us; error 27 us; skew 500 ppm
Sep 29 16:31:09 dist-test-slave-mlmz krb5kdc[26746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699469, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:09.883217 26987 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:31:09.884027 26987 webserver.cc:460] Webserver started at http://127.23.44.2:39529/ using document root <none> and password file <none>
I20260929 16:31:09.884572 26987 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:09.884670 26987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:09.884927 26987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:09.886935 26987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "986faf28459f4e148c6c2766b4248d69"
format_stamp: "Formatted at 2026-09-29 16:31:09 on dist-test-slave-mlmz"
server_key: "a2b950e0b4a66721ecdd3b842485c8fb"
server_key_iv: "d523223b781ef1745704ec53268e8c4c"
server_key_version: "encryptionkey@0"
I20260929 16:31:09.887497 26987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "986faf28459f4e148c6c2766b4248d69"
format_stamp: "Formatted at 2026-09-29 16:31:09 on dist-test-slave-mlmz"
server_key: "a2b950e0b4a66721ecdd3b842485c8fb"
server_key_iv: "d523223b781ef1745704ec53268e8c4c"
server_key_version: "encryptionkey@0"
I20260929 16:31:09.891546 26987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:31:09.893991 27002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:09.895186 26987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:09.895323 26987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "986faf28459f4e148c6c2766b4248d69"
format_stamp: "Formatted at 2026-09-29 16:31:09 on dist-test-slave-mlmz"
server_key: "a2b950e0b4a66721ecdd3b842485c8fb"
server_key_iv: "d523223b781ef1745704ec53268e8c4c"
server_key_version: "encryptionkey@0"
I20260929 16:31:09.895442 26987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:09.906049 26987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:09.908962 26987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:09.909153 26987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:09.909826 26987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:09.910593 26987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:09.910669 26987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:09.910737 26987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:09.910770 26987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:09.919066 26987 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:44531
I20260929 16:31:09.919091 27115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:44531 every 8 connection(s)
I20260929 16:31:09.919893 26987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:09.928254 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 26987
I20260929 16:31:09.928452 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:09.928838 23728 external_mini_cluster.cc:1428] Setting key 88937aca9e8c4d0bc6f711ae0eafe2d1
Sep 29 16:31:09 dist-test-slave-mlmz krb5kdc[26746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699469, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:09.935093 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:09.921712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:42859 (local address 127.23.44.62:39115)
0929 16:31:09.921858 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:09.921862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:09.922814 (+ 952us) server_negotiation.cc:407] Connection header received
0929 16:31:09.924240 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:09.924243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:09.924290 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:09.924376 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:09.925621 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:09.926186 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:09.927316 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:09.927541 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:09.927635 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:09.931016 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:09.931019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:09.931021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:09.931044 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:09.932649 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:09.933343 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:09.933346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:09.933348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:09.933422 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:09.933837 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:09.933840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:09.933842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:09.934053 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:09.934140 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:09.934835 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 16:31:09.934963 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:31:09.936271 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:09.922146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39115 (local address 127.23.44.2:42859)
0929 16:31:09.922664 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:09.922704 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:09.924029 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:09.924552 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:09.924558 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:09.924910 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:09.925451 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:09.925466 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:09.926306 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:09.926309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:09.927197 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:09.927206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:09.927697 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:09.927700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:09.927859 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:09.928594 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:09.928615 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:09.930823 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:09.932768 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:09.932775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:09.932793 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:09.933164 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:09.933521 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:09.933527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:09.933532 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:09.933718 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:09.934169 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:09.934179 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:09.934520 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 16:31:09.934740 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 16:31:09.935125 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":111,"threads_started":1}
I20260929 16:31:09.937578 27116 heartbeater.cc:347] Connected to a master server at 127.23.44.62:39115
I20260929 16:31:09.937819 27116 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:09.938308 27116 heartbeater.cc:511] Master 127.23.44.62:39115 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:31:09.939384 26790 ts_manager.cc:194] Registered new tserver with Master: 986faf28459f4e148c6c2766b4248d69 (127.23.44.2:44531)
I20260929 16:31:09.939944 26790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:42859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:31:09.956321 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39115
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:10.064376 27121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:10.064670 27121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:10.064742 27121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:10.068806 27121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:10.068897 27121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:10.069003 27121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:10.074496 27121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:10.075429 27121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:10.076383 27121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:10.085659 27127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:10.085680 27129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:10.085799 27121 server_base.cc:1034] running on GCE node
W20260929 16:31:10.085958 27126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:10.086413 27121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:10.087023 27121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:10.088210 27121 hybrid_clock.cc:648] HybridClock initialized: now 1790699470088177 us; error 45 us; skew 500 ppm
Sep 29 16:31:10 dist-test-slave-mlmz krb5kdc[26746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699470, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:10.090924 27121 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:31:10.091805 27121 webserver.cc:460] Webserver started at http://127.23.44.3:39907/ using document root <none> and password file <none>
I20260929 16:31:10.092345 27121 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:10.092435 27121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:10.092655 27121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:10.094664 27121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "d9b689ac8dd84ac98cc559f01f9c64aa"
format_stamp: "Formatted at 2026-09-29 16:31:10 on dist-test-slave-mlmz"
server_key: "d6e8cf3b0edd87931650e9d2c8285edf"
server_key_iv: "9c141d30beaf5f97227061d89f9d4119"
server_key_version: "encryptionkey@0"
I20260929 16:31:10.095203 27121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "d9b689ac8dd84ac98cc559f01f9c64aa"
format_stamp: "Formatted at 2026-09-29 16:31:10 on dist-test-slave-mlmz"
server_key: "d6e8cf3b0edd87931650e9d2c8285edf"
server_key_iv: "9c141d30beaf5f97227061d89f9d4119"
server_key_version: "encryptionkey@0"
I20260929 16:31:10.099404 27121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:31:10.101867 27136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:10.103070 27121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:10.103214 27121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "d9b689ac8dd84ac98cc559f01f9c64aa"
format_stamp: "Formatted at 2026-09-29 16:31:10 on dist-test-slave-mlmz"
server_key: "d6e8cf3b0edd87931650e9d2c8285edf"
server_key_iv: "9c141d30beaf5f97227061d89f9d4119"
server_key_version: "encryptionkey@0"
I20260929 16:31:10.103323 27121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:10.143359 27121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:10.146644 27121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:10.146881 27121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:10.147495 27121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:10.148353 27121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:10.148432 27121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:10.148509 27121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:10.148558 27121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:10.158114 27121 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:42315
I20260929 16:31:10.158157 27249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:42315 every 8 connection(s)
I20260929 16:31:10.159008 27121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:10.162439 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 27121
I20260929 16:31:10.162530 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:10.162827 23728 external_mini_cluster.cc:1428] Setting key fcc2e51124f7adb93c7ac3f8e20274f5
Sep 29 16:31:10 dist-test-slave-mlmz krb5kdc[26746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699470, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:10.173346 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:10.160858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:44799 (local address 127.23.44.62:39115)
0929 16:31:10.161012 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:10.161016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:10.161908 (+ 892us) server_negotiation.cc:407] Connection header received
0929 16:31:10.163536 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:10.163540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:10.163595 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:10.163698 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:10.165072 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:10.165614 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:10.166758 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:10.167036 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:10.167131 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:10.169497 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:10.169501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:10.169503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:10.169525 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:10.171105 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:10.171624 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:10.171627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:10.171628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:10.171706 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:10.172233 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:10.172236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:10.172237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:10.172414 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:10.172501 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:10.173060 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 16:31:10.173158 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:31:10.174173 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:10.161222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39115 (local address 127.23.44.3:44799)
0929 16:31:10.161748 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:10.161790 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:10.163295 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:10.163884 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:10.163891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:10.164345 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:10.164940 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:10.164957 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:10.165742 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:10.165745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:10.166632 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:10.166641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:10.167171 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:10.167174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:10.167316 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:10.167901 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:10.167916 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:10.169342 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:10.171219 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:10.171232 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:10.171244 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:10.171526 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:10.171982 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:10.171985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:10.171987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:10.172117 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:10.172549 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:10.172555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:10.172793 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 16:31:10.173002 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 16:31:10.173245 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":113,"threads_started":1}
I20260929 16:31:10.175374 27250 heartbeater.cc:347] Connected to a master server at 127.23.44.62:39115
I20260929 16:31:10.175669 27250 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:10.176261 27250 heartbeater.cc:511] Master 127.23.44.62:39115 requested a full tablet report, sending...
I20260929 16:31:10.177353 26790 ts_manager.cc:194] Registered new tserver with Master: d9b689ac8dd84ac98cc559f01f9c64aa (127.23.44.3:42315)
I20260929 16:31:10.177862 26790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:44799
I20260929 16:31:10.188287 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:31:10 dist-test-slave-mlmz krb5kdc[26746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699470, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 16:31:10 dist-test-slave-mlmz krb5kdc[26746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699470, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:10.215068 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:10.205357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.23.44.62:39115)
0929 16:31:10.205499 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:10.205503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:10.205653 (+ 150us) server_negotiation.cc:407] Connection header received
0929 16:31:10.205761 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:10.205764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:10.205807 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:10.205888 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:10.206698 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:10.207490 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:10.208526 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:10.208772 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:10.208866 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:10.211642 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:10.211647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:10.211650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:10.211683 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:10.213250 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:10.213750 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:10.213754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:10.213756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:10.213836 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:10.214134 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:10.214137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:10.214139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:10.214364 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:10.214517 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:10.214764 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 16:31:10.214885 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 16:31:10.215444 26790 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58812
I20260929 16:31:10.216500 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 26853
I20260929 16:31:10.222522 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 26987
I20260929 16:31:10.228546 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 27121
I20260929 16:31:10.234112 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 26759
2026-09-29T16:31:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2155 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:10 dist-test-slave-mlmz krb5kdc[27262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:10 dist-test-slave-mlmz krb5kdc[27262](info): set up 2 sockets
Sep 29 16:31:10 dist-test-slave-mlmz krb5kdc[27262](info): commencing operation
krb5kdc: starting...
W20260929 16:31:11.269145 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:31:11 dist-test-slave-mlmz krb5kdc[27262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:31:11.388729 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:37001
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:37001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:11.492422 27275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:11.492695 27275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:11.492766 27275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:11.496718 27275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:11.496811 27275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:11.496848 27275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:11.496908 27275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:11.496954 27275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:11.502192 27275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:37001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:37001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:11.503120 27275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:11.504043 27275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:11.512180 27280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:11.512184 27281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:11.512250 27283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:11.512219 27275 server_base.cc:1034] running on GCE node
I20260929 16:31:11.512961 27275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:11.514026 27275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:11.515198 27275 hybrid_clock.cc:648] HybridClock initialized: now 1790699471515178 us; error 32 us; skew 500 ppm
Sep 29 16:31:11 dist-test-slave-mlmz krb5kdc[27262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699471, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:11.517802 27275 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:31:11.518599 27275 webserver.cc:460] Webserver started at http://127.23.44.62:40659/ using document root <none> and password file <none>
I20260929 16:31:11.519116 27275 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:11.519182 27275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:11.519377 27275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:11.521250 27275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "bb8c8fbf1682401d84bd04d9bb2d35ca"
format_stamp: "Formatted at 2026-09-29 16:31:11 on dist-test-slave-mlmz"
server_key: "7967f99a23f47dcbe49b154fa42faffd"
server_key_iv: "77730f55cd3b8529f61c73ef7f1812e1"
server_key_version: "encryptionkey@0"
I20260929 16:31:11.521831 27275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "bb8c8fbf1682401d84bd04d9bb2d35ca"
format_stamp: "Formatted at 2026-09-29 16:31:11 on dist-test-slave-mlmz"
server_key: "7967f99a23f47dcbe49b154fa42faffd"
server_key_iv: "77730f55cd3b8529f61c73ef7f1812e1"
server_key_version: "encryptionkey@0"
I20260929 16:31:11.525344 27275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:31:11.527518 27290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:11.528874 27275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 16:31:11.529057 27275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "bb8c8fbf1682401d84bd04d9bb2d35ca"
format_stamp: "Formatted at 2026-09-29 16:31:11 on dist-test-slave-mlmz"
server_key: "7967f99a23f47dcbe49b154fa42faffd"
server_key_iv: "77730f55cd3b8529f61c73ef7f1812e1"
server_key_version: "encryptionkey@0"
I20260929 16:31:11.529178 27275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:11.552944 27275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:11.556143 27275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:11.556373 27275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:11.564234 27275 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:37001
I20260929 16:31:11.564262 27342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:37001 every 8 connection(s)
I20260929 16:31:11.565194 27275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:11.567843 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:11.572925 27343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca: Bootstrap starting.
I20260929 16:31:11.573944 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 27275
I20260929 16:31:11.574055 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:11.574344 23728 external_mini_cluster.cc:1428] Setting key 534dd3b009de57e1ceb13f658e0585d7
I20260929 16:31:11.575575 27343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:11.576318 27343 log.cc:824] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:11.578182 27343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca: No bootstrap required, opened a new log
I20260929 16:31:11.582531 27343 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8c8fbf1682401d84bd04d9bb2d35ca" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 37001 } }
I20260929 16:31:11.582768 27343 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:11.582852 27343 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8c8fbf1682401d84bd04d9bb2d35ca, State: Initialized, Role: FOLLOWER
I20260929 16:31:11.583262 27343 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8c8fbf1682401d84bd04d9bb2d35ca" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 37001 } }
I20260929 16:31:11.583431 27343 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:11.583509 27343 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:11.583612 27343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [term 0 FOLLOWER]: Advancing to term 1
Sep 29 16:31:11 dist-test-slave-mlmz krb5kdc[27262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:11.584643 27343 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8c8fbf1682401d84bd04d9bb2d35ca" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 37001 } }
I20260929 16:31:11.585018 27343 leader_election.cc:302] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb8c8fbf1682401d84bd04d9bb2d35ca; no voters:
I20260929 16:31:11.585402 27343 leader_election.cc:288] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:11.585634 27348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:11.585857 27348 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [term 1 LEADER]: Becoming Leader. State: Replica: bb8c8fbf1682401d84bd04d9bb2d35ca, State: Running, Role: LEADER
I20260929 16:31:11.586231 27348 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8c8fbf1682401d84bd04d9bb2d35ca" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 37001 } }
I20260929 16:31:11.586758 27343 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:11.587848 27349 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb8c8fbf1682401d84bd04d9bb2d35ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8c8fbf1682401d84bd04d9bb2d35ca" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 37001 } } }
I20260929 16:31:11.588057 27349 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:11.588406 27350 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb8c8fbf1682401d84bd04d9bb2d35ca. Latest consensus state: current_term: 1 leader_uuid: "bb8c8fbf1682401d84bd04d9bb2d35ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8c8fbf1682401d84bd04d9bb2d35ca" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 37001 } } }
I20260929 16:31:11.588585 27350 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:11.590451 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:11.575453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.23.44.62:37001)
0929 16:31:11.575900 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:11.575915 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:11.575973 (+ 58us) server_negotiation.cc:407] Connection header received
0929 16:31:11.576696 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:11.576720 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:11.577166 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:11.577564 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:11.578421 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:11.579948 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:11.580881 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:11.581680 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:11.581848 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:11.584791 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:11.584797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:11.584813 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:11.584841 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:11.586958 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:11.587520 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:11.587526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:11.587532 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:11.587671 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:11.588000 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:11.588006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:11.588011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:11.588495 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:11.588671 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:11.589456 (+ 785us) server_negotiation.cc:299] Negotiation successful
0929 16:31:11.589761 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":122,"threads_started":1}
W20260929 16:31:11.591826 27363 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:31:11.591913 27363 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:31:11.591982 27357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:11.592828 27357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:11.598031 27357 catalog_manager.cc:1347] Generated new cluster ID: 023513084bd24698a8d738d13d74d4fd
I20260929 16:31:11.598094 27357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:11.603760 27357 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:11.605014 27357 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:11.609714 27357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb8c8fbf1682401d84bd04d9bb2d35ca: Generated new TSK 0
I20260929 16:31:11.610375 27357 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:31:11.640514 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:37001
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:11.752928 27369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:11.753161 27369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:11.753202 27369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:11.757243 27369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:11.757344 27369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:11.757445 27369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:11.762459 27369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:37001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:11.763384 27369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:11.764313 27369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:11.773943 27375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:11.773975 27377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:11.774109 27369 server_base.cc:1034] running on GCE node
W20260929 16:31:11.773947 27374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:11.774534 27369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:11.775149 27369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:11.776327 27369 hybrid_clock.cc:648] HybridClock initialized: now 1790699471776312 us; error 27 us; skew 500 ppm
Sep 29 16:31:11 dist-test-slave-mlmz krb5kdc[27262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699471, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:11.778923 27369 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:31:11.779687 27369 webserver.cc:460] Webserver started at http://127.23.44.1:41707/ using document root <none> and password file <none>
I20260929 16:31:11.780205 27369 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:11.780292 27369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:11.780515 27369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:11.782490 27369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "68a94bebc4f14f8cb6ba0b20e51473c5"
format_stamp: "Formatted at 2026-09-29 16:31:11 on dist-test-slave-mlmz"
server_key: "db951be3f44a34914ec57a729bc4bf4e"
server_key_iv: "44feba48776a3d813da6611dbd5663fc"
server_key_version: "encryptionkey@0"
I20260929 16:31:11.783022 27369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "68a94bebc4f14f8cb6ba0b20e51473c5"
format_stamp: "Formatted at 2026-09-29 16:31:11 on dist-test-slave-mlmz"
server_key: "db951be3f44a34914ec57a729bc4bf4e"
server_key_iv: "44feba48776a3d813da6611dbd5663fc"
server_key_version: "encryptionkey@0"
I20260929 16:31:11.786494 27369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:31:11.788950 27384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:11.790187 27369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:11.790315 27369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "68a94bebc4f14f8cb6ba0b20e51473c5"
format_stamp: "Formatted at 2026-09-29 16:31:11 on dist-test-slave-mlmz"
server_key: "db951be3f44a34914ec57a729bc4bf4e"
server_key_iv: "44feba48776a3d813da6611dbd5663fc"
server_key_version: "encryptionkey@0"
I20260929 16:31:11.790426 27369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:11.813724 27369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:11.816924 27369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:11.817121 27369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:11.817734 27369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:11.818495 27369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:11.818571 27369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:11.818647 27369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:11.818684 27369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:11.827476 27369 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:37311
I20260929 16:31:11.827502 27497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:37311 every 8 connection(s)
I20260929 16:31:11.828274 27369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:11.837085 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 27369
I20260929 16:31:11.837222 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
Sep 29 16:31:11 dist-test-slave-mlmz krb5kdc[27262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699471, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:11.837554 23728 external_mini_cluster.cc:1428] Setting key f1bf31c9de601ebb64ef5058b1ee9564
I20260929 16:31:11.841634 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:11.830069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:48197 (local address 127.23.44.62:37001)
0929 16:31:11.830198 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:11.830202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:11.830934 (+ 732us) server_negotiation.cc:407] Connection header received
0929 16:31:11.832313 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:11.832316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:11.832362 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:11.832489 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:11.833636 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:11.834236 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:11.835307 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:11.835547 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:11.835631 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:11.838207 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:11.838211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:11.838214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:11.838239 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:11.839654 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:11.840162 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:11.840168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:11.840171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:11.840250 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:11.840582 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:11.840587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:11.840589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:11.840759 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:11.840867 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:11.841347 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 16:31:11.841471 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:31:11.842410 27500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:11.830304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:37001 (local address 127.23.44.1:48197)
0929 16:31:11.830755 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:11.830794 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:11.832101 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:11.832624 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:11.832630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:11.832959 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:11.833496 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:11.833510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:11.834343 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:11.834345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:11.835195 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:11.835203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:11.835680 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:11.835683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:11.835815 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:11.836314 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:11.836328 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:11.838019 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:11.839774 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:11.839780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:11.839793 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:11.840055 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:11.840343 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:11.840347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:11.840349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:11.840486 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:11.840870 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:11.840879 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:11.841104 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 16:31:11.841302 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 16:31:11.841599 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260929 16:31:11.843765 27498 heartbeater.cc:347] Connected to a master server at 127.23.44.62:37001
I20260929 16:31:11.843999 27498 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:11.844445 27498 heartbeater.cc:511] Master 127.23.44.62:37001 requested a full tablet report, sending...
I20260929 16:31:11.846060 27306 ts_manager.cc:194] Registered new tserver with Master: 68a94bebc4f14f8cb6ba0b20e51473c5 (127.23.44.1:37311)
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
I20260929 16:31:11.847664 27306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:48197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:31:11.866707 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:37001
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:11.973496 27503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:11.973719 27503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:11.973791 27503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:11.977813 27503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:11.977866 27503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:11.977943 27503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:11.983119 27503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:37001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:11.984087 27503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:11.985069 27503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:31:11.994628 27503 server_base.cc:1034] running on GCE node
W20260929 16:31:11.994594 27511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:11.994611 27508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:11.994592 27509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:11.995177 27503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:11.995775 27503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:11.997012 27503 hybrid_clock.cc:648] HybridClock initialized: now 1790699471996994 us; error 28 us; skew 500 ppm
Sep 29 16:31:11 dist-test-slave-mlmz krb5kdc[27262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699471, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:11.999783 27503 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:31:12.000710 27503 webserver.cc:460] Webserver started at http://127.23.44.2:38287/ using document root <none> and password file <none>
I20260929 16:31:12.001353 27503 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:12.001451 27503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:12.001734 27503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:12.003865 27503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "1984b728cbf3469685c66cb0f55d6c75"
format_stamp: "Formatted at 2026-09-29 16:31:12 on dist-test-slave-mlmz"
server_key: "8f004f0d0039bd161b606a8cc7a85693"
server_key_iv: "2da0316851f35349064866ad885914f6"
server_key_version: "encryptionkey@0"
I20260929 16:31:12.004447 27503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "1984b728cbf3469685c66cb0f55d6c75"
format_stamp: "Formatted at 2026-09-29 16:31:12 on dist-test-slave-mlmz"
server_key: "8f004f0d0039bd161b606a8cc7a85693"
server_key_iv: "2da0316851f35349064866ad885914f6"
server_key_version: "encryptionkey@0"
I20260929 16:31:12.008217 27503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 16:31:12.010685 27518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:12.011936 27503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:31:12.012060 27503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "1984b728cbf3469685c66cb0f55d6c75"
format_stamp: "Formatted at 2026-09-29 16:31:12 on dist-test-slave-mlmz"
server_key: "8f004f0d0039bd161b606a8cc7a85693"
server_key_iv: "2da0316851f35349064866ad885914f6"
server_key_version: "encryptionkey@0"
I20260929 16:31:12.012161 27503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:12.025210 27503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:12.028031 27503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:12.028223 27503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:12.028745 27503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:12.029560 27503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:12.029630 27503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:12.029699 27503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:12.029752 27503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:12.038811 27503 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:38219
I20260929 16:31:12.038853 27631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:38219 every 8 connection(s)
I20260929 16:31:12.039645 27503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:12.042454 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 27503
I20260929 16:31:12.042589 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:12.042863 23728 external_mini_cluster.cc:1428] Setting key a52a65272a13973c314a40a6ed827cb9
Sep 29 16:31:12 dist-test-slave-mlmz krb5kdc[27262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699471, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:31:12.053536 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:12.041320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:38773 (local address 127.23.44.62:37001)
0929 16:31:12.041438 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:12.041441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:12.042231 (+ 790us) server_negotiation.cc:407] Connection header received
0929 16:31:12.043676 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:12.043680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:12.043730 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:12.043833 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:12.044861 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:12.045441 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:12.046551 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:12.046962 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:12.047081 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:12.049700 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:12.049705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:12.049708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:12.049736 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:12.051537 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:12.052025 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:12.052032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:12.052036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:12.052132 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:12.052517 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:12.052519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:12.052520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:12.052680 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:12.052758 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:12.053314 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 16:31:12.053425 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
I20260929 16:31:12.054453 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:12.041651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:37001 (local address 127.23.44.2:38773)
0929 16:31:12.042068 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:12.042117 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:12.043469 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:12.043994 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:12.044001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:12.044308 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:12.044726 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:12.044738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:12.045557 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:12.045566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:12.046428 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:12.046437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:12.047106 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:12.047109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:12.047244 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:12.047831 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:12.047855 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:12.049533 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:12.051652 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:12.051657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:12.051667 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:12.051906 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:12.052258 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:12.052261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:12.052263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:12.052413 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:12.052775 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:12.052780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:12.053034 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 16:31:12.053198 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 16:31:12.053517 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":104,"threads_started":1}
I20260929 16:31:12.055737 27632 heartbeater.cc:347] Connected to a master server at 127.23.44.62:37001
I20260929 16:31:12.055971 27632 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:12.056474 27632 heartbeater.cc:511] Master 127.23.44.62:37001 requested a full tablet report, sending...
I20260929 16:31:12.057483 27306 ts_manager.cc:194] Registered new tserver with Master: 1984b728cbf3469685c66cb0f55d6c75 (127.23.44.2:38219)
I20260929 16:31:12.058305 27306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:38773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:31:12.072121 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:37001
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:12.177896 27637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:12.178115 27637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:12.178156 27637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:12.182049 27637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:12.182092 27637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:12.182147 27637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:12.187136 27637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:37001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:12.188030 27637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:12.188944 27637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:12.198060 27645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:12.198114 27637 server_base.cc:1034] running on GCE node
W20260929 16:31:12.198071 27643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:12.198071 27642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:12.198637 27637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:12.199151 27637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:12.200349 27637 hybrid_clock.cc:648] HybridClock initialized: now 1790699472200316 us; error 24 us; skew 500 ppm
Sep 29 16:31:12 dist-test-slave-mlmz krb5kdc[27262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699472, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:12.203411 27637 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:31:12.204201 27637 webserver.cc:460] Webserver started at http://127.23.44.3:35359/ using document root <none> and password file <none>
I20260929 16:31:12.204712 27637 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:12.204772 27637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:12.204931 27637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:12.206981 27637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "67fdef78f4b24955a08f56a29d160eb9"
format_stamp: "Formatted at 2026-09-29 16:31:12 on dist-test-slave-mlmz"
server_key: "4857d62b4b2d661bb5dc7cc404d211ba"
server_key_iv: "8ec71bd3c293c885eb3f787a5282231b"
server_key_version: "encryptionkey@0"
I20260929 16:31:12.207531 27637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "67fdef78f4b24955a08f56a29d160eb9"
format_stamp: "Formatted at 2026-09-29 16:31:12 on dist-test-slave-mlmz"
server_key: "4857d62b4b2d661bb5dc7cc404d211ba"
server_key_iv: "8ec71bd3c293c885eb3f787a5282231b"
server_key_version: "encryptionkey@0"
I20260929 16:31:12.211213 27637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 16:31:12.213662 27652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:12.214890 27637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:12.215001 27637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "67fdef78f4b24955a08f56a29d160eb9"
format_stamp: "Formatted at 2026-09-29 16:31:12 on dist-test-slave-mlmz"
server_key: "4857d62b4b2d661bb5dc7cc404d211ba"
server_key_iv: "8ec71bd3c293c885eb3f787a5282231b"
server_key_version: "encryptionkey@0"
I20260929 16:31:12.215098 27637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:12.228453 27637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:12.231236 27637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:12.231428 27637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:12.231957 27637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:12.232740 27637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:12.232785 27637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:12.232821 27637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:12.232880 27637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:12.241859 27637 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:37213
I20260929 16:31:12.241904 27765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:37213 every 8 connection(s)
I20260929 16:31:12.242686 27637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:12.247854 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 27637
I20260929 16:31:12.247949 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:12.248211 23728 external_mini_cluster.cc:1428] Setting key 627dfc0161074c319ff656ee2ef83b90
Sep 29 16:31:12 dist-test-slave-mlmz krb5kdc[27262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699472, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:12.256143 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:12.244382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:46247 (local address 127.23.44.62:37001)
0929 16:31:12.244529 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:12.244533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:12.245328 (+ 795us) server_negotiation.cc:407] Connection header received
0929 16:31:12.246832 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:12.246836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:12.246898 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:12.247022 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:12.248302 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:12.248961 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:12.250506 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:12.250755 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:12.250876 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:12.253172 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:12.253175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:12.253177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:12.253198 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:12.254494 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:12.254929 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:12.254932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:12.254933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:12.255009 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:12.255279 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:12.255281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:12.255282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:12.255435 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:12.255513 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:12.255909 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 16:31:12.256033 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:31:12.256880 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:12.244717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:37001 (local address 127.23.44.3:46247)
0929 16:31:12.245155 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:12.245196 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:12.246602 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:12.247190 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:12.247197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:12.247574 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:12.248144 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:12.248159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:12.249098 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:12.249102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:12.250369 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:12.250378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:12.250908 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:12.250912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:12.251095 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:12.251637 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:12.251653 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:12.253024 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:12.254594 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:12.254598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:12.254608 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:12.254845 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:12.255086 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:12.255088 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:12.255089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:12.255205 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:12.255519 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:12.255524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:12.255720 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 16:31:12.255870 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 16:31:12.256094 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":109,"threads_started":1}
I20260929 16:31:12.257931 27766 heartbeater.cc:347] Connected to a master server at 127.23.44.62:37001
I20260929 16:31:12.258147 27766 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:12.258569 27766 heartbeater.cc:511] Master 127.23.44.62:37001 requested a full tablet report, sending...
I20260929 16:31:12.259486 27306 ts_manager.cc:194] Registered new tserver with Master: 67fdef78f4b24955a08f56a29d160eb9 (127.23.44.3:37213)
I20260929 16:31:12.259989 27306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:46247
I20260929 16:31:12.262108 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:31:12 dist-test-slave-mlmz krb5kdc[27262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:31:12 dist-test-slave-mlmz krb5kdc[27262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:31:12.287981 27773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:12.278185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.23.44.1:37311)
0929 16:31:12.278501 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:12.278511 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:12.278535 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:31:12.278583 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:12.278588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:12.278757 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:12.278873 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:12.279528 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:12.280300 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:12.281132 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:12.281568 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:12.281656 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:12.284214 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:12.284223 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:12.284234 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:12.284256 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:12.285970 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:12.286391 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:12.286397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:12.286398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:12.286488 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:12.286774 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:12.286777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:12.286778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:12.287037 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:12.287187 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:12.287506 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 16:31:12.287798 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
Sep 29 16:31:12 dist-test-slave-mlmz krb5kdc[27262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:12.299592 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:12.291028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.23.44.62:37001)
0929 16:31:12.291168 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:12.291174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:12.291253 (+ 79us) server_negotiation.cc:407] Connection header received
0929 16:31:12.291382 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:12.291385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:12.291426 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:12.291510 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:12.292115 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:12.292628 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:12.293471 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:12.293798 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:12.293906 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:12.296458 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:12.296461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:12.296463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:12.296485 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:12.297964 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:12.298426 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:12.298429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:12.298430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:12.298517 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:12.298802 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:12.298805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:12.298806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:12.299043 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:12.299164 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:12.299371 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 16:31:12.299471 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 16:31:12.300014 27306 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53980
I20260929 16:31:12.300849 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 27369
I20260929 16:31:12.308096 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 27503
I20260929 16:31:12.313918 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 27637
I20260929 16:31:12.319077 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 27275
2026-09-29T16:31:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2082 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:12 dist-test-slave-mlmz krb5kdc[27778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:12 dist-test-slave-mlmz krb5kdc[27778](info): set up 2 sockets
Sep 29 16:31:12 dist-test-slave-mlmz krb5kdc[27778](info): commencing operation
krb5kdc: starting...
W20260929 16:31:13.405328 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:31:13 dist-test-slave-mlmz krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:31:13.524873 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:34777
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:34777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:13.650271 27791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:13.650552 27791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:13.650626 27791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:13.654563 27791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:13.654641 27791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:13.654678 27791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:13.654732 27791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:13.654771 27791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:13.659919 27791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:34777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:34777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:13.660868 27791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:13.661821 27791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:13.669569 27796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:13.669595 27797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:13.669659 27799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:13.669878 27791 server_base.cc:1034] running on GCE node
I20260929 16:31:13.670383 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:13.671380 27791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:13.672565 27791 hybrid_clock.cc:648] HybridClock initialized: now 1790699473672550 us; error 38 us; skew 500 ppm
Sep 29 16:31:13 dist-test-slave-mlmz krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699473, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:13.675282 27791 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:31:13.675998 27791 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:13.676093 27791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:13.676316 27791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:13.678315 27791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "d3c5615000744f5ab335252897efbbd6"
format_stamp: "Formatted at 2026-09-29 16:31:13 on dist-test-slave-mlmz"
server_key: "d2bf4a9e72a6c6e4f40e0a0d8e09a232"
server_key_iv: "a89fdb33731e93109586ccf9f90c8c4f"
server_key_version: "encryptionkey@0"
I20260929 16:31:13.678866 27791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "d3c5615000744f5ab335252897efbbd6"
format_stamp: "Formatted at 2026-09-29 16:31:13 on dist-test-slave-mlmz"
server_key: "d2bf4a9e72a6c6e4f40e0a0d8e09a232"
server_key_iv: "a89fdb33731e93109586ccf9f90c8c4f"
server_key_version: "encryptionkey@0"
I20260929 16:31:13.682282 27791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 16:31:13.684535 27805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:13.685712 27791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:31:13.685848 27791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "d3c5615000744f5ab335252897efbbd6"
format_stamp: "Formatted at 2026-09-29 16:31:13 on dist-test-slave-mlmz"
server_key: "d2bf4a9e72a6c6e4f40e0a0d8e09a232"
server_key_iv: "a89fdb33731e93109586ccf9f90c8c4f"
server_key_version: "encryptionkey@0"
I20260929 16:31:13.685951 27791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:13.704229 27791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:13.707252 27791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:13.707438 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:13.714231 27791 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:34777
I20260929 16:31:13.714504 27857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:34777 every 8 connection(s)
I20260929 16:31:13.714866 27791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:13.717492 27858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:13.720533 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 27791
I20260929 16:31:13.720686 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:13.720938 23728 external_mini_cluster.cc:1428] Setting key f89560b4588ceccede242027a4238818
I20260929 16:31:13.723196 27858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6: Bootstrap starting.
I20260929 16:31:13.725829 27858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:13.726668 27858 log.cc:824] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:13.728551 27858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6: No bootstrap required, opened a new log
Sep 29 16:31:13 dist-test-slave-mlmz krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:13.732774 27858 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c5615000744f5ab335252897efbbd6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34777 } }
I20260929 16:31:13.733032 27858 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:13.733139 27858 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c5615000744f5ab335252897efbbd6, State: Initialized, Role: FOLLOWER
I20260929 16:31:13.733634 27858 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c5615000744f5ab335252897efbbd6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34777 } }
I20260929 16:31:13.733757 27858 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:13.733839 27858 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:13.733940 27858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:13.734771 27858 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c5615000744f5ab335252897efbbd6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34777 } }
I20260929 16:31:13.735149 27858 leader_election.cc:302] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c5615000744f5ab335252897efbbd6; no voters:
I20260929 16:31:13.735466 27858 leader_election.cc:288] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:13.735569 27863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:13.735759 27863 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [term 1 LEADER]: Becoming Leader. State: Replica: d3c5615000744f5ab335252897efbbd6, State: Running, Role: LEADER
I20260929 16:31:13.736047 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:13.722265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42216 (local address 127.23.44.62:34777)
0929 16:31:13.722649 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:13.722667 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:13.722708 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:31:13.723516 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:13.723538 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:13.723924 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:13.724293 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:13.724883 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:13.726386 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:13.727343 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:13.727799 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:13.727984 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:13.730762 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:13.730767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:13.730779 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:13.730804 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:13.732839 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:13.733388 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:13.733394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:13.733398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:13.733516 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:13.733843 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:13.733847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:13.733849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:13.734301 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:13.734433 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:13.734837 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 16:31:13.735112 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":94,"threads_started":1}
I20260929 16:31:13.736059 27863 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c5615000744f5ab335252897efbbd6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34777 } }
I20260929 16:31:13.736724 27858 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:13.738082 27865 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c5615000744f5ab335252897efbbd6. Latest consensus state: current_term: 1 leader_uuid: "d3c5615000744f5ab335252897efbbd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c5615000744f5ab335252897efbbd6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34777 } } }
I20260929 16:31:13.738202 27865 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:13.738586 27864 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c5615000744f5ab335252897efbbd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c5615000744f5ab335252897efbbd6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34777 } } }
I20260929 16:31:13.738731 27864 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6 [sys.catalog]: This master's current role is: LEADER
W20260929 16:31:13.741197 27878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:31:13.741303 27878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:31:13.741385 27873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:13.742038 27873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:13.747332 27873 catalog_manager.cc:1347] Generated new cluster ID: d818a61da0a445c6b1b334e08c3dcf88
I20260929 16:31:13.747393 27873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:13.753506 27873 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:13.754320 27873 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:13.765712 27873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6: Generated new TSK 0
I20260929 16:31:13.766310 27873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:31:13.803925 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34777
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:13.925802 27884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:13.926030 27884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:13.926071 27884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:13.930037 27884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:13.930085 27884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:13.930171 27884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:13.935189 27884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:13.936148 27884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:13.937093 27884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:13.946406 27889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:13.946398 27890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:13.946470 27892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:13.946863 27884 server_base.cc:1034] running on GCE node
I20260929 16:31:13.947191 27884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:13.947759 27884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:13.948966 27884 hybrid_clock.cc:648] HybridClock initialized: now 1790699473948937 us; error 52 us; skew 500 ppm
Sep 29 16:31:13 dist-test-slave-mlmz krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699473, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:13.951831 27884 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:31:13.952523 27884 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:13.952617 27884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:13.952838 27884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:13.954851 27884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "67812d55792f4e34824076d49031c98b"
format_stamp: "Formatted at 2026-09-29 16:31:13 on dist-test-slave-mlmz"
server_key: "89e5e8a5ee2f25510da600e12c6cb268"
server_key_iv: "ffeeb45f4bf9c58067d57bf9f48098b1"
server_key_version: "encryptionkey@0"
I20260929 16:31:13.955400 27884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "67812d55792f4e34824076d49031c98b"
format_stamp: "Formatted at 2026-09-29 16:31:13 on dist-test-slave-mlmz"
server_key: "89e5e8a5ee2f25510da600e12c6cb268"
server_key_iv: "ffeeb45f4bf9c58067d57bf9f48098b1"
server_key_version: "encryptionkey@0"
I20260929 16:31:13.959080 27884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 16:31:13.961516 27898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:13.962658 27884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:13.962792 27884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "67812d55792f4e34824076d49031c98b"
format_stamp: "Formatted at 2026-09-29 16:31:13 on dist-test-slave-mlmz"
server_key: "89e5e8a5ee2f25510da600e12c6cb268"
server_key_iv: "ffeeb45f4bf9c58067d57bf9f48098b1"
server_key_version: "encryptionkey@0"
I20260929 16:31:13.962929 27884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:13.988296 27884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:13.991814 27884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:13.992026 27884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:13.992616 27884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:13.993481 27884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:13.993551 27884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:13.993615 27884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:13.993667 27884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:14.002476 27884 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:38923
I20260929 16:31:14.002494 28011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:38923 every 8 connection(s)
I20260929 16:31:14.003044 27884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:14.010105 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 27884
I20260929 16:31:14.010236 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:14.010519 23728 external_mini_cluster.cc:1428] Setting key a3cfc28fc4050f7b278c2acb06469842
Sep 29 16:31:14 dist-test-slave-mlmz krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699473, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:14.016736 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.004718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:52369 (local address 127.23.44.62:34777)
0929 16:31:14.004862 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:14.004867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:14.006030 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 16:31:14.007005 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:14.007008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:14.007054 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:14.007132 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:14.008260 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.008814 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.009983 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.010283 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.010401 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.012915 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:14.012919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:14.012922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:14.012951 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:14.014687 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:14.015207 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:14.015211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:14.015213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:14.015302 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:14.015650 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:14.015653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:14.015655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:14.015835 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:14.015950 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:14.016445 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 16:31:14.016570 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:31:14.017537 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.005115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:34777 (local address 127.23.44.1:52369)
0929 16:31:14.005893 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:14.005931 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:14.006778 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:14.007285 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:14.007292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:14.007631 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:14.008079 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.008092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.008936 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.008938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.009854 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.009863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.010440 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.010444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.010652 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.011231 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:14.011253 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:14.012754 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:14.014798 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:14.014804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:14.014818 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:14.015102 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:14.015401 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:14.015404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:14.015406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:14.015546 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:14.015948 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:14.015955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:14.016201 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 16:31:14.016383 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 16:31:14.016685 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260929 16:31:14.018846 28012 heartbeater.cc:347] Connected to a master server at 127.23.44.62:34777
I20260929 16:31:14.019085 28012 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:14.019516 28012 heartbeater.cc:511] Master 127.23.44.62:34777 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
I20260929 16:31:14.021015 27822 ts_manager.cc:194] Registered new tserver with Master: 67812d55792f4e34824076d49031c98b (127.23.44.1:38923)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
I20260929 16:31:14.022323 27822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:52369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:31:14.036340 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34777
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:14.141389 28017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:14.141606 28017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:14.141647 28017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:14.145437 28017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:14.145479 28017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:14.145535 28017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:14.150667 28017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:14.151608 28017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:14.152515 28017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:14.161291 28025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:14.161320 28022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:14.161312 28023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:14.161435 28017 server_base.cc:1034] running on GCE node
I20260929 16:31:14.161877 28017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:14.162487 28017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:14.163671 28017 hybrid_clock.cc:648] HybridClock initialized: now 1790699474163656 us; error 23 us; skew 500 ppm
Sep 29 16:31:14 dist-test-slave-mlmz krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699474, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:14.166410 28017 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:31:14.167163 28017 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:14.167223 28017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:14.167428 28017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:14.169371 28017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "3231b8035729485abedd42edb016167f"
format_stamp: "Formatted at 2026-09-29 16:31:14 on dist-test-slave-mlmz"
server_key: "958c7bae873362d5048889874ec003cc"
server_key_iv: "45cdf82410ce4b3dd29ba4b61f4aa79f"
server_key_version: "encryptionkey@0"
I20260929 16:31:14.169911 28017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "3231b8035729485abedd42edb016167f"
format_stamp: "Formatted at 2026-09-29 16:31:14 on dist-test-slave-mlmz"
server_key: "958c7bae873362d5048889874ec003cc"
server_key_iv: "45cdf82410ce4b3dd29ba4b61f4aa79f"
server_key_version: "encryptionkey@0"
I20260929 16:31:14.173405 28017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:31:14.175628 28031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:14.176790 28017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:14.176918 28017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "3231b8035729485abedd42edb016167f"
format_stamp: "Formatted at 2026-09-29 16:31:14 on dist-test-slave-mlmz"
server_key: "958c7bae873362d5048889874ec003cc"
server_key_iv: "45cdf82410ce4b3dd29ba4b61f4aa79f"
server_key_version: "encryptionkey@0"
I20260929 16:31:14.177026 28017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:14.197817 28017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:14.200604 28017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:14.200804 28017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:14.201401 28017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:14.202181 28017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:14.202253 28017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:14.202315 28017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:14.202358 28017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:14.211433 28017 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:43659
I20260929 16:31:14.211457 28144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:43659 every 8 connection(s)
I20260929 16:31:14.211984 28017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
Sep 29 16:31:14 dist-test-slave-mlmz krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699474, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:14.222203 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 28017
I20260929 16:31:14.222379 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:14.222775 23728 external_mini_cluster.cc:1428] Setting key bfa65184ad1948ff2ea2a3ad64ea29e6
I20260929 16:31:14.225461 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.213692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:47347 (local address 127.23.44.62:34777)
0929 16:31:14.213858 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:14.213862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:14.215005 (+ 1143us) server_negotiation.cc:407] Connection header received
0929 16:31:14.215804 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:14.215807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:14.215851 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:14.215935 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:14.217027 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.217655 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.218703 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.218911 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.219013 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.221361 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:14.221365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:14.221367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:14.221391 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:14.223246 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:14.223790 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:14.223796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:14.223798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:14.223889 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:14.224301 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:14.224304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:14.224306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:14.224500 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:14.224611 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:14.225122 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 16:31:14.225252 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:31:14.226352 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.214076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:34777 (local address 127.23.44.2:47347)
0929 16:31:14.214827 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:14.214866 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:14.215615 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:14.216081 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:14.216087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:14.216415 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:14.216865 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.216877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.217760 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.217762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.218602 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.218611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.219056 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.219059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.219187 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.219707 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:14.219722 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:14.221158 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:14.223374 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:14.223380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:14.223395 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:14.223674 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:14.224023 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:14.224030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:14.224032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:14.224178 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:14.224623 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:14.224631 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:14.224882 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 16:31:14.225077 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 16:31:14.225424 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":102,"threads_started":1}
I20260929 16:31:14.227507 28145 heartbeater.cc:347] Connected to a master server at 127.23.44.62:34777
I20260929 16:31:14.227797 28145 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:14.228253 28145 heartbeater.cc:511] Master 127.23.44.62:34777 requested a full tablet report, sending...
I20260929 16:31:14.229362 27822 ts_manager.cc:194] Registered new tserver with Master: 3231b8035729485abedd42edb016167f (127.23.44.2:43659)
I20260929 16:31:14.230176 27822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:47347
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:31:14.249465 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34777
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:14.355667 28150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:14.355901 28150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:14.355943 28150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:14.359944 28150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:14.359994 28150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:14.360087 28150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:14.365052 28150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:14.366230 28150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:14.367471 28150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:14.376696 28155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:14.376715 28158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:14.376694 28156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:14.376876 28150 server_base.cc:1034] running on GCE node
I20260929 16:31:14.377426 28150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:14.377949 28150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:14.379139 28150 hybrid_clock.cc:648] HybridClock initialized: now 1790699474379122 us; error 26 us; skew 500 ppm
Sep 29 16:31:14 dist-test-slave-mlmz krb5kdc[27778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699474, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:14.382025 28150 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:31:14.382750 28150 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:14.382812 28150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:14.383042 28150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:14.384935 28150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "4bb76555ef364b0383b221a1149d30da"
format_stamp: "Formatted at 2026-09-29 16:31:14 on dist-test-slave-mlmz"
server_key: "600ec36599863d41d0f2b5860221190c"
server_key_iv: "1fdbb42d1fc98fc938ce5d070dd99bf0"
server_key_version: "encryptionkey@0"
I20260929 16:31:14.385493 28150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "4bb76555ef364b0383b221a1149d30da"
format_stamp: "Formatted at 2026-09-29 16:31:14 on dist-test-slave-mlmz"
server_key: "600ec36599863d41d0f2b5860221190c"
server_key_iv: "1fdbb42d1fc98fc938ce5d070dd99bf0"
server_key_version: "encryptionkey@0"
I20260929 16:31:14.388787 28150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 16:31:14.390995 28164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:14.392131 28150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:14.392247 28150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "4bb76555ef364b0383b221a1149d30da"
format_stamp: "Formatted at 2026-09-29 16:31:14 on dist-test-slave-mlmz"
server_key: "600ec36599863d41d0f2b5860221190c"
server_key_iv: "1fdbb42d1fc98fc938ce5d070dd99bf0"
server_key_version: "encryptionkey@0"
I20260929 16:31:14.392345 28150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:14.406342 28150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:14.409421 28150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:14.409624 28150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:14.410177 28150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:14.410951 28150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:14.411026 28150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:14.411089 28150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:14.411126 28150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:14.419836 28150 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:34253
I20260929 16:31:14.419862 28277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:34253 every 8 connection(s)
I20260929 16:31:14.420392 28150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:14.425010 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 28150
I20260929 16:31:14.425148 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:14.425510 23728 external_mini_cluster.cc:1428] Setting key 4a24e94fb3ac176bfad89fac280b3326
Sep 29 16:31:14 dist-test-slave-mlmz krb5kdc[27778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699474, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:14.433904 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.422179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:48653 (local address 127.23.44.62:34777)
0929 16:31:14.422323 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:14.422327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:14.423058 (+ 731us) server_negotiation.cc:407] Connection header received
0929 16:31:14.424341 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:14.424344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:14.424389 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:14.424463 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:14.426038 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.426775 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.428140 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.428345 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.428424 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.430768 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:14.430771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:14.430773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:14.430795 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:14.432001 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:14.432517 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:14.432520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:14.432521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:14.432595 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:14.432894 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:14.432896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:14.432897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:14.433067 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:14.433143 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:14.433701 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 16:31:14.433792 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:31:14.434614 28280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.422488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:34777 (local address 127.23.44.3:48653)
0929 16:31:14.422902 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:14.422945 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:14.424158 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:14.424633 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:14.424640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:14.425156 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:14.425872 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.425894 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.426928 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.426931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.428032 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.428042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.428479 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.428482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.428617 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.429116 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:14.429132 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:14.430625 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:14.432122 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:14.432127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:14.432138 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:14.432410 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:14.432678 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:14.432681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:14.432682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:14.432811 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:14.433178 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:14.433183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:14.433482 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 16:31:14.433656 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 16:31:14.433880 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":99,"threads_started":1}
I20260929 16:31:14.435609 28278 heartbeater.cc:347] Connected to a master server at 127.23.44.62:34777
I20260929 16:31:14.435822 28278 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:14.436264 28278 heartbeater.cc:511] Master 127.23.44.62:34777 requested a full tablet report, sending...
I20260929 16:31:14.437213 27822 ts_manager.cc:194] Registered new tserver with Master: 4bb76555ef364b0383b221a1149d30da (127.23.44.3:34253)
I20260929 16:31:14.437880 27822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:48653
I20260929 16:31:14.439167 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:31:14.448417 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.441116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.23.44.62:34777)
0929 16:31:14.441251 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:14.441279 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:14.441482 (+ 203us) server_negotiation.cc:407] Connection header received
0929 16:31:14.441520 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:14.441522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:14.441557 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:14.441631 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:14.442305 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.442783 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.443608 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.443811 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.443897 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.445473 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:14.445476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:14.445477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:14.445497 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:14.446680 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:14.447117 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:14.447120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:14.447121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:14.447196 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:14.447509 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:14.447512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:14.447513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:14.447713 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:14.447832 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:14.448115 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 16:31:14.448268 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:31:14.451110 27821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:14.453977 27821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:14.478264 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.471327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:38923 (local address 127.0.0.1:59712)
0929 16:31:14.471639 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:14.471682 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:14.471798 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:14.473088 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:14.473094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:14.473121 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:14.473294 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.473302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.474828 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.474832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.476301 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.476312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.477676 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.477681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.477800 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.477839 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 16:31:14.477974 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 16:31:14.478100 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260929 16:31:14.478863 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.471723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43659 (local address 127.0.0.1:48312)
0929 16:31:14.473179 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:14.473193 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:14.473303 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:14.474013 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:14.474016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:14.474034 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:14.474117 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.474122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.475625 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.475628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.476878 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.476888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.478235 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.478239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.478348 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.478574 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 16:31:14.478652 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:31:14.478721 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1402,"thread_start_us":82,"threads_started":1}
I20260929 16:31:14.478870 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.472053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.23.44.1:38923)
0929 16:31:14.472354 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:14.472361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:14.472386 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:31:14.472467 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:14.472472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:14.472709 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:14.472874 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:14.473435 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.474673 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.476444 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.477363 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.477467 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.477609 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:14.478622 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0929 16:31:14.478721 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260929 16:31:14.479589 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.472102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.23.44.2:43659)
0929 16:31:14.473049 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:14.473063 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:14.473422 (+ 359us) server_negotiation.cc:407] Connection header received
0929 16:31:14.473515 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:14.473521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:14.473706 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:14.473851 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:14.474245 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.475465 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.477019 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.477942 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.478044 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.478174 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:14.479304 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0929 16:31:14.479383 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":134,"threads_started":1}
I20260929 16:31:14.480247 28291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.471423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:34253 (local address 127.0.0.1:44118)
0929 16:31:14.472659 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:14.472674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:14.472783 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:14.474168 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:14.474173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:14.474193 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:14.474307 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.474314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.475842 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.475846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.477990 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.478005 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.479432 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.479436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.479568 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.479591 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:31:14.479770 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 16:31:14.479862 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"thread_start_us":102,"threads_started":1}
I20260929 16:31:14.481030 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.472105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.23.44.3:34253)
0929 16:31:14.472555 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:14.472570 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:14.472774 (+ 204us) server_negotiation.cc:407] Connection header received
0929 16:31:14.473642 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:14.473650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:14.473862 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:14.474011 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:14.474436 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.475684 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.478155 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.479302 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.480483 (+ 1181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.480629 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:14.480677 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 16:31:14.480845 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":160,"threads_started":1}
I20260929 16:31:14.482056 27946 tablet_service.cc:1467] Processing CreateTablet for tablet e7a81f9c952d47ed9bd7853a68ccfe83 (DEFAULT_TABLE table=test-table [id=83123491b5c04bf680efd48fcf84dd1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:14.483220 27946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a81f9c952d47ed9bd7853a68ccfe83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:14.483482 28079 tablet_service.cc:1467] Processing CreateTablet for tablet e7a81f9c952d47ed9bd7853a68ccfe83 (DEFAULT_TABLE table=test-table [id=83123491b5c04bf680efd48fcf84dd1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:14.484535 28079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a81f9c952d47ed9bd7853a68ccfe83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:14.487346 28212 tablet_service.cc:1467] Processing CreateTablet for tablet e7a81f9c952d47ed9bd7853a68ccfe83 (DEFAULT_TABLE table=test-table [id=83123491b5c04bf680efd48fcf84dd1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:14.488322 28212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a81f9c952d47ed9bd7853a68ccfe83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:14.490629 28295 tablet_bootstrap.cc:492] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b: Bootstrap starting.
I20260929 16:31:14.492898 28295 tablet_bootstrap.cc:654] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:14.494098 28295 log.cc:824] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:14.494706 28296 tablet_bootstrap.cc:492] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da: Bootstrap starting.
I20260929 16:31:14.494971 28298 tablet_bootstrap.cc:492] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f: Bootstrap starting.
I20260929 16:31:14.496616 28295 tablet_bootstrap.cc:492] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b: No bootstrap required, opened a new log
I20260929 16:31:14.496797 28295 ts_tablet_manager.cc:1397] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:31:14.497022 28296 tablet_bootstrap.cc:654] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:14.497027 28298 tablet_bootstrap.cc:654] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:14.498219 28298 log.cc:824] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:14.498832 28296 log.cc:824] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:14.500530 28295 raft_consensus.cc:348] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67812d55792f4e34824076d49031c98b" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38923 } } peers { permanent_uuid: "4bb76555ef364b0383b221a1149d30da" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34253 } } peers { permanent_uuid: "3231b8035729485abedd42edb016167f" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43659 } }
I20260929 16:31:14.500759 28295 raft_consensus.cc:374] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:14.500774 28298 tablet_bootstrap.cc:492] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f: No bootstrap required, opened a new log
I20260929 16:31:14.500809 28295 raft_consensus.cc:729] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67812d55792f4e34824076d49031c98b, State: Initialized, Role: FOLLOWER
I20260929 16:31:14.500952 28298 ts_tablet_manager.cc:1397] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:31:14.501310 28295 consensus_queue.cc:260] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67812d55792f4e34824076d49031c98b" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38923 } } peers { permanent_uuid: "4bb76555ef364b0383b221a1149d30da" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34253 } } peers { permanent_uuid: "3231b8035729485abedd42edb016167f" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43659 } }
I20260929 16:31:14.502125 28296 tablet_bootstrap.cc:492] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da: No bootstrap required, opened a new log
I20260929 16:31:14.502262 28295 ts_tablet_manager.cc:1428] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 16:31:14.502291 28296 ts_tablet_manager.cc:1397] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 16:31:14.502594 28012 heartbeater.cc:503] Master 127.23.44.62:34777 was elected leader, sending a full tablet report...
W20260929 16:31:14.504035 28013 tablet.cc:2369] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:31:14.506441 28298 raft_consensus.cc:348] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67812d55792f4e34824076d49031c98b" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38923 } } peers { permanent_uuid: "4bb76555ef364b0383b221a1149d30da" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34253 } } peers { permanent_uuid: "3231b8035729485abedd42edb016167f" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43659 } }
I20260929 16:31:14.506722 28298 raft_consensus.cc:374] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:14.506798 28298 raft_consensus.cc:729] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3231b8035729485abedd42edb016167f, State: Initialized, Role: FOLLOWER
I20260929 16:31:14.507009 28296 raft_consensus.cc:348] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67812d55792f4e34824076d49031c98b" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38923 } } peers { permanent_uuid: "4bb76555ef364b0383b221a1149d30da" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34253 } } peers { permanent_uuid: "3231b8035729485abedd42edb016167f" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43659 } }
I20260929 16:31:14.507278 28296 raft_consensus.cc:374] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:14.507339 28296 raft_consensus.cc:729] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb76555ef364b0383b221a1149d30da, State: Initialized, Role: FOLLOWER
I20260929 16:31:14.507336 28298 consensus_queue.cc:260] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67812d55792f4e34824076d49031c98b" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38923 } } peers { permanent_uuid: "4bb76555ef364b0383b221a1149d30da" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34253 } } peers { permanent_uuid: "3231b8035729485abedd42edb016167f" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43659 } }
I20260929 16:31:14.507792 28296 consensus_queue.cc:260] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67812d55792f4e34824076d49031c98b" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38923 } } peers { permanent_uuid: "4bb76555ef364b0383b221a1149d30da" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34253 } } peers { permanent_uuid: "3231b8035729485abedd42edb016167f" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43659 } }
I20260929 16:31:14.507964 28145 heartbeater.cc:503] Master 127.23.44.62:34777 was elected leader, sending a full tablet report...
I20260929 16:31:14.508330 28298 ts_tablet_manager.cc:1428] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:31:14.508697 28296 ts_tablet_manager.cc:1428] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:31:14.508890 28278 heartbeater.cc:503] Master 127.23.44.62:34777 was elected leader, sending a full tablet report...
I20260929 16:31:14.510562 28302 raft_consensus.cc:482] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:14.511029 28302 raft_consensus.cc:504] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67812d55792f4e34824076d49031c98b" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38923 } } peers { permanent_uuid: "4bb76555ef364b0383b221a1149d30da" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34253 } } peers { permanent_uuid: "3231b8035729485abedd42edb016167f" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43659 } }
I20260929 16:31:14.512092 28302 leader_election.cc:288] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67812d55792f4e34824076d49031c98b (127.23.44.1:38923), 4bb76555ef364b0383b221a1149d30da (127.23.44.3:34253)
I20260929 16:31:14.516638 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.512418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:48517 (local address 127.23.44.3:34253)
0929 16:31:14.512557 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:14.512562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:14.512855 (+ 293us) server_negotiation.cc:407] Connection header received
0929 16:31:14.513066 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:14.513071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:14.513119 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:14.513172 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:14.513845 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.514377 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.515417 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.516035 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.516140 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.516215 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:14.516440 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 16:31:14.516513 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:31:14.516817 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.512265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:38923 (local address 127.23.44.2:35443)
0929 16:31:14.512406 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:14.512421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:14.512502 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:14.512876 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:14.512879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:14.512895 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:14.513027 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.513034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.513985 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.513991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.515434 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.515447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.516343 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.516346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.516482 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.516503 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:31:14.516572 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:31:14.516636 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"spinlock_wait_cycles":27392}
I20260929 16:31:14.516948 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.512583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:35443 (local address 127.23.44.1:38923)
0929 16:31:14.512693 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:14.512702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:14.512712 (+ 10us) server_negotiation.cc:407] Connection header received
0929 16:31:14.512744 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:14.512746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:14.512780 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:14.512866 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:14.513153 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.513878 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.515562 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.516148 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.516240 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.516286 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:14.516740 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 16:31:14.516824 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:31:14.516686 28304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.512265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:34253 (local address 127.23.44.2:48517)
0929 16:31:14.512770 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:14.512784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:14.512945 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:14.513570 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:14.513576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:14.513597 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:14.513731 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.513740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.514494 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.514497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.515302 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:14.515310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.516157 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.516164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:14.516297 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.516337 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:31:14.516415 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:31:14.516495 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":162,"spinlock_wait_cycles":768,"thread_start_us":86,"threads_started":1}
I20260929 16:31:14.517479 28232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a81f9c952d47ed9bd7853a68ccfe83" candidate_uuid: "3231b8035729485abedd42edb016167f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb76555ef364b0383b221a1149d30da" is_pre_election: true
I20260929 16:31:14.517474 27966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a81f9c952d47ed9bd7853a68ccfe83" candidate_uuid: "3231b8035729485abedd42edb016167f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67812d55792f4e34824076d49031c98b" is_pre_election: true
I20260929 16:31:14.517717 27966 raft_consensus.cc:2454] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3231b8035729485abedd42edb016167f in term 0.
I20260929 16:31:14.517717 28232 raft_consensus.cc:2454] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3231b8035729485abedd42edb016167f in term 0.
I20260929 16:31:14.518157 28034 leader_election.cc:302] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3231b8035729485abedd42edb016167f, 67812d55792f4e34824076d49031c98b; no voters:
I20260929 16:31:14.518390 28302 raft_consensus.cc:2781] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:14.518482 28302 raft_consensus.cc:482] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:14.518529 28302 raft_consensus.cc:3037] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:14.519462 28302 raft_consensus.cc:504] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67812d55792f4e34824076d49031c98b" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38923 } } peers { permanent_uuid: "4bb76555ef364b0383b221a1149d30da" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34253 } } peers { permanent_uuid: "3231b8035729485abedd42edb016167f" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43659 } }
I20260929 16:31:14.519790 28302 leader_election.cc:288] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [CANDIDATE]: Term 1 election: Requested vote from peers 67812d55792f4e34824076d49031c98b (127.23.44.1:38923), 4bb76555ef364b0383b221a1149d30da (127.23.44.3:34253)
I20260929 16:31:14.520076 28232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a81f9c952d47ed9bd7853a68ccfe83" candidate_uuid: "3231b8035729485abedd42edb016167f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb76555ef364b0383b221a1149d30da"
I20260929 16:31:14.520192 28232 raft_consensus.cc:3037] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:14.520233 27966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a81f9c952d47ed9bd7853a68ccfe83" candidate_uuid: "3231b8035729485abedd42edb016167f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67812d55792f4e34824076d49031c98b"
I20260929 16:31:14.520328 27966 raft_consensus.cc:3037] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:14.521416 28232 raft_consensus.cc:2454] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3231b8035729485abedd42edb016167f in term 1.
I20260929 16:31:14.521478 27966 raft_consensus.cc:2454] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3231b8035729485abedd42edb016167f in term 1.
I20260929 16:31:14.521710 28032 leader_election.cc:302] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3231b8035729485abedd42edb016167f, 4bb76555ef364b0383b221a1149d30da; no voters:
I20260929 16:31:14.521857 28302 raft_consensus.cc:2781] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:14.522053 28302 raft_consensus.cc:686] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 1 LEADER]: Becoming Leader. State: Replica: 3231b8035729485abedd42edb016167f, State: Running, Role: LEADER
I20260929 16:31:14.522336 28302 consensus_queue.cc:237] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67812d55792f4e34824076d49031c98b" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38923 } } peers { permanent_uuid: "4bb76555ef364b0383b221a1149d30da" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34253 } } peers { permanent_uuid: "3231b8035729485abedd42edb016167f" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43659 } }
I20260929 16:31:14.525326 27821 catalog_manager.cc:5495] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3231b8035729485abedd42edb016167f (127.23.44.2). New cstate: current_term: 1 leader_uuid: "3231b8035729485abedd42edb016167f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67812d55792f4e34824076d49031c98b" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bb76555ef364b0383b221a1149d30da" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3231b8035729485abedd42edb016167f" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43659 } health_report { overall_health: HEALTHY } } }
I20260929 16:31:14.544198 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.540743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.23.44.2:43659)
0929 16:31:14.540880 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:14.540885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:14.540899 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:31:14.541042 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:14.541045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:14.541104 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:14.541202 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:14.541474 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.541973 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.542810 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:14.543041 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:14.543148 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:14.543455 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:14.543825 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:14.543951 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:14.544009 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 16:31:14.544061 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:31:14.550482 27966 raft_consensus.cc:1261] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b [term 1 FOLLOWER]: Refusing update from remote peer 3231b8035729485abedd42edb016167f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:14.550787 28232 raft_consensus.cc:1261] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da [term 1 FOLLOWER]: Refusing update from remote peer 3231b8035729485abedd42edb016167f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:14.551208 28302 consensus_queue.cc:1035] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67812d55792f4e34824076d49031c98b" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:14.551425 28305 consensus_queue.cc:1035] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bb76555ef364b0383b221a1149d30da" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:14.557747 28308 mvcc.cc:205] Tried to move back new op lower bound from 7334705047750758400 to 7334705047645753344. Current Snapshot: MvccSnapshot[applied={T|T < 7334705047750758400}]
I20260929 16:31:14.566221 27822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:14.566354 27822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:14.568531 27822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3c5615000744f5ab335252897efbbd6: Sending DeleteTablet for 3 replicas of tablet e7a81f9c952d47ed9bd7853a68ccfe83
I20260929 16:31:14.569178 27946 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a81f9c952d47ed9bd7853a68ccfe83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:14 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:59712
I20260929 16:31:14.569214 28212 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a81f9c952d47ed9bd7853a68ccfe83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:14 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:44118
I20260929 16:31:14.569667 28318 tablet_replica.cc:330] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b: stopping tablet replica
I20260929 16:31:14.569689 28319 tablet_replica.cc:330] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da: stopping tablet replica
I20260929 16:31:14.569962 28319 raft_consensus.cc:2229] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:14.570015 28079 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a81f9c952d47ed9bd7853a68ccfe83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:14 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:48312
I20260929 16:31:14.570106 28318 raft_consensus.cc:2229] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:14.570232 28319 raft_consensus.cc:2258] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:14.570604 28318 raft_consensus.cc:2258] T e7a81f9c952d47ed9bd7853a68ccfe83 P 67812d55792f4e34824076d49031c98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:14.570911 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 27884
I20260929 16:31:14.572602 28319 ts_tablet_manager.cc:1905] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:14.573112 28320 tablet_replica.cc:330] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f: stopping tablet replica
I20260929 16:31:14.573385 28320 raft_consensus.cc:2229] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:14.573704 28320 raft_consensus.cc:2258] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:14.574901 28320 ts_tablet_manager.cc:1905] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:14.575562 28319 ts_tablet_manager.cc:1918] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:14.575655 28319 log.cc:1196] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/e7a81f9c952d47ed9bd7853a68ccfe83
I20260929 16:31:14.576006 28319 ts_tablet_manager.cc:1939] T e7a81f9c952d47ed9bd7853a68ccfe83 P 4bb76555ef364b0383b221a1149d30da: Deleting consensus metadata
I20260929 16:31:14.577091 27806 catalog_manager.cc:4841] TS 4bb76555ef364b0383b221a1149d30da (127.23.44.3:34253): tablet e7a81f9c952d47ed9bd7853a68ccfe83 (table test-table [id=83123491b5c04bf680efd48fcf84dd1e]) successfully deleted
I20260929 16:31:14.578130 28320 ts_tablet_manager.cc:1918] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:14.578225 28320 log.cc:1196] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/e7a81f9c952d47ed9bd7853a68ccfe83
I20260929 16:31:14.578549 28320 ts_tablet_manager.cc:1939] T e7a81f9c952d47ed9bd7853a68ccfe83 P 3231b8035729485abedd42edb016167f: Deleting consensus metadata
W20260929 16:31:14.579384 27808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:38923 (error 108)
I20260929 16:31:14.579833 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 28017
I20260929 16:31:14.581168 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.580083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:38923 (local address 127.0.0.1:59728)
0929 16:31:14.581043 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:14.581103 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":887}
W20260929 16:31:14.581342 27808 catalog_manager.cc:4584] TS 67812d55792f4e34824076d49031c98b (127.23.44.1:38923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7a81f9c952d47ed9bd7853a68ccfe83: Network error: Client connection negotiation failed: client connection to 127.23.44.1:38923: connect: Connection refused (error 111)
I20260929 16:31:14.587265 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 28150
I20260929 16:31:14.587450 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.587198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43659 (local address 127.0.0.1:48340)
0929 16:31:14.587291 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:14.587363 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.2:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 16:31:14.592245 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 27791
2026-09-29T16:31:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2273 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T16:31:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:14Z Disabled control of system clock
I20260929 16:31:14.621156 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:38271
--webserver_interface=127.23.44.62
--webserver_port=0
--builtin_ntp_servers=127.23.44.20:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:38271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:31:14.728863 28325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:14.729089 28325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:14.729123 28325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:14.732992 28325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:14.733042 28325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:14.733057 28325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:14.733073 28325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:14.733124 28325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:14.737805 28325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:38271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:38271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:14.738745 28325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:14.740028 28325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:14.747947 28330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:14.747987 28333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:14.747982 28331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:14.747963 28325 server_base.cc:1034] running on GCE node
I20260929 16:31:14.748733 28325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:14.749852 28325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:14.751034 28325 hybrid_clock.cc:648] HybridClock initialized: now 1790699474751006 us; error 39 us; skew 500 ppm
I20260929 16:31:14.752777 28325 webserver.cc:460] Webserver started at http://127.23.44.62:36745/ using document root <none> and password file <none>
I20260929 16:31:14.753371 28325 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:14.753468 28325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:14.753688 28325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:14.755599 28325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "43bea382fece45439fd8a06fbda01aac"
format_stamp: "Formatted at 2026-09-29 16:31:14 on dist-test-slave-mlmz"
server_key: "865f96cd1d8a6b10d2e412c4a6348486"
server_key_iv: "0bfde4fa51faf7f2eb697ccb4e51b2e5"
server_key_version: "encryptionkey@0"
I20260929 16:31:14.756146 28325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "43bea382fece45439fd8a06fbda01aac"
format_stamp: "Formatted at 2026-09-29 16:31:14 on dist-test-slave-mlmz"
server_key: "865f96cd1d8a6b10d2e412c4a6348486"
server_key_iv: "0bfde4fa51faf7f2eb697ccb4e51b2e5"
server_key_version: "encryptionkey@0"
I20260929 16:31:14.759624 28325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:31:14.761978 28339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:14.763145 28325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:14.763278 28325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "43bea382fece45439fd8a06fbda01aac"
format_stamp: "Formatted at 2026-09-29 16:31:14 on dist-test-slave-mlmz"
server_key: "865f96cd1d8a6b10d2e412c4a6348486"
server_key_iv: "0bfde4fa51faf7f2eb697ccb4e51b2e5"
server_key_version: "encryptionkey@0"
I20260929 16:31:14.763386 28325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:14.767691 28325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:14.768229 28325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:14.768400 28325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:14.775254 28325 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:38271
I20260929 16:31:14.775288 28391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:38271 every 8 connection(s)
I20260929 16:31:14.776117 28325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:14.776454 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 28325
I20260929 16:31:14.776597 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:14.776894 23728 external_mini_cluster.cc:1428] Setting key ac75bce737a0413af8ce38ee8c1eaeac
I20260929 16:31:14.778998 28392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:14.782685 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:14.778191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.23.44.62:38271)
0929 16:31:14.779294 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:14.779312 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:14.779347 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:31:14.779999 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:14.780014 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:14.780344 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:14.780618 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:14.781477 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:14.781482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:14.781493 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:14.781504 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:14.781558 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:14.781578 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:14.781688 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:14.781811 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 16:31:14.782052 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":605,"threads_started":1}
I20260929 16:31:14.784596 28392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac: Bootstrap starting.
I20260929 16:31:14.786615 28392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:14.787309 28392 log.cc:824] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:14.789059 28392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac: No bootstrap required, opened a new log
I20260929 16:31:14.792240 28392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43bea382fece45439fd8a06fbda01aac" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38271 } }
I20260929 16:31:14.792467 28392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:14.792558 28392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bea382fece45439fd8a06fbda01aac, State: Initialized, Role: FOLLOWER
I20260929 16:31:14.792944 28392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43bea382fece45439fd8a06fbda01aac" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38271 } }
I20260929 16:31:14.793097 28392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:14.793180 28392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:14.793310 28392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:14.794116 28392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43bea382fece45439fd8a06fbda01aac" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38271 } }
I20260929 16:31:14.794435 28392 leader_election.cc:302] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43bea382fece45439fd8a06fbda01aac; no voters:
I20260929 16:31:14.794718 28392 leader_election.cc:288] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:14.794852 28397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:14.795073 28397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [term 1 LEADER]: Becoming Leader. State: Replica: 43bea382fece45439fd8a06fbda01aac, State: Running, Role: LEADER
I20260929 16:31:14.795363 28397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43bea382fece45439fd8a06fbda01aac" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38271 } }
I20260929 16:31:14.795946 28392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:14.797109 28398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43bea382fece45439fd8a06fbda01aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43bea382fece45439fd8a06fbda01aac" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38271 } } }
I20260929 16:31:14.797140 28399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43bea382fece45439fd8a06fbda01aac. Latest consensus state: current_term: 1 leader_uuid: "43bea382fece45439fd8a06fbda01aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43bea382fece45439fd8a06fbda01aac" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38271 } } }
I20260929 16:31:14.797247 28398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:14.797281 28399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:14.797641 28406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:14.800467 28406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:14.805155 28406 catalog_manager.cc:1347] Generated new cluster ID: da7fe13beb97406fa90f73f825088e6c
I20260929 16:31:14.805217 28406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:14.823693 28406 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:14.823985 28406 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:14.835815 28406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac: Generated new TSK 0
I20260929 16:31:14.836377 28406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:31:14.839000 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38271
--builtin_ntp_servers=127.23.44.20:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:31:14.962757 28416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:14.962986 28416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:14.963023 28416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:14.967020 28416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:14.967074 28416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:14.967129 28416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:14.971644 28416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:14.972529 28416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:14.973732 28416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:14.982852 28421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:14.982950 28416 server_base.cc:1034] running on GCE node
W20260929 16:31:14.982903 28422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:14.982903 28424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:14.983449 28416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:14.984117 28416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:14.985319 28416 hybrid_clock.cc:648] HybridClock initialized: now 1790699474985310 us; error 31 us; skew 500 ppm
I20260929 16:31:14.987222 28416 webserver.cc:460] Webserver started at http://127.23.44.1:41533/ using document root <none> and password file <none>
I20260929 16:31:14.987773 28416 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:14.987833 28416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:14.988037 28416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:14.990016 28416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "164e7a61b23a4e439af7c0bc9b1543d4"
format_stamp: "Formatted at 2026-09-29 16:31:14 on dist-test-slave-mlmz"
server_key: "16947a86ce35b904ec1fcb2380991e9d"
server_key_iv: "3f335e52bc49fc141c50022a2adfa50a"
server_key_version: "encryptionkey@0"
I20260929 16:31:14.990537 28416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "164e7a61b23a4e439af7c0bc9b1543d4"
format_stamp: "Formatted at 2026-09-29 16:31:14 on dist-test-slave-mlmz"
server_key: "16947a86ce35b904ec1fcb2380991e9d"
server_key_iv: "3f335e52bc49fc141c50022a2adfa50a"
server_key_version: "encryptionkey@0"
I20260929 16:31:14.994035 28416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:31:14.996388 28430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:14.997649 28416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:31:14.997783 28416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "164e7a61b23a4e439af7c0bc9b1543d4"
format_stamp: "Formatted at 2026-09-29 16:31:14 on dist-test-slave-mlmz"
server_key: "16947a86ce35b904ec1fcb2380991e9d"
server_key_iv: "3f335e52bc49fc141c50022a2adfa50a"
server_key_version: "encryptionkey@0"
I20260929 16:31:14.997888 28416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:15.002106 28416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:15.002643 28416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:15.002815 28416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:15.003321 28416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:15.004136 28416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:15.004204 28416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:15.004269 28416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:15.004318 28416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:15.012792 28416 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:43389
I20260929 16:31:15.012930 28543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:43389 every 8 connection(s)
I20260929 16:31:15.013695 28416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:15.015241 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 28416
I20260929 16:31:15.015406 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:15.015681 23728 external_mini_cluster.cc:1428] Setting key 3cbe50ace41f932ec635e109aab334b7
I20260929 16:31:15.017745 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38271
--builtin_ntp_servers=127.23.44.20:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 16:31:15.020637 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.015497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:34521 (local address 127.23.44.62:38271)
0929 16:31:15.015658 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:15.015664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:15.016819 (+ 1155us) server_negotiation.cc:407] Connection header received
0929 16:31:15.018677 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:15.018680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:15.018713 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:15.018777 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:15.020131 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:15.020137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:15.020141 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:15.020155 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:15.020175 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:15.020187 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:15.020279 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:15.020507 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 16:31:15.020543 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:31:15.021775 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.015938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38271 (local address 127.23.44.1:34521)
0929 16:31:15.016657 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:15.016709 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:15.018431 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:15.018988 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:15.019000 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:15.019123 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:15.019813 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:15.019815 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:15.019868 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:15.019955 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:15.020290 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:15.020299 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:15.020305 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:31:15.020516 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 16:31:15.020754 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":135,"threads_started":1}
I20260929 16:31:15.023082 28544 heartbeater.cc:347] Connected to a master server at 127.23.44.62:38271
I20260929 16:31:15.023317 28544 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:15.023703 28544 heartbeater.cc:511] Master 127.23.44.62:38271 requested a full tablet report, sending...
I20260929 16:31:15.025079 28356 ts_manager.cc:194] Registered new tserver with Master: 164e7a61b23a4e439af7c0bc9b1543d4 (127.23.44.1:43389)
W20260929 16:31:15.125576 28547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:15.125790 28547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:15.125825 28547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:15.129784 28547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:15.129833 28547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:15.129885 28547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:15.134466 28547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:15.135426 28547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:15.136723 28547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:15.145671 28552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:15.145706 28553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:15.145830 28555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:15.145879 28547 server_base.cc:1034] running on GCE node
I20260929 16:31:15.146327 28547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:15.146895 28547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:15.148079 28547 hybrid_clock.cc:648] HybridClock initialized: now 1790699475148061 us; error 27 us; skew 500 ppm
I20260929 16:31:15.149932 28547 webserver.cc:460] Webserver started at http://127.23.44.2:43639/ using document root <none> and password file <none>
I20260929 16:31:15.150435 28547 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:15.150493 28547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:15.150647 28547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:15.152498 28547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "cf23551073614f3f8931300433c7d6ca"
format_stamp: "Formatted at 2026-09-29 16:31:15 on dist-test-slave-mlmz"
server_key: "2f9fadffaafc304c453eb25c7169c860"
server_key_iv: "6fa084006c8523f8b9cfc0baad7cf7ae"
server_key_version: "encryptionkey@0"
I20260929 16:31:15.152977 28547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "cf23551073614f3f8931300433c7d6ca"
format_stamp: "Formatted at 2026-09-29 16:31:15 on dist-test-slave-mlmz"
server_key: "2f9fadffaafc304c453eb25c7169c860"
server_key_iv: "6fa084006c8523f8b9cfc0baad7cf7ae"
server_key_version: "encryptionkey@0"
I20260929 16:31:15.156435 28547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:31:15.158862 28561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:15.159987 28547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:31:15.160081 28547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "cf23551073614f3f8931300433c7d6ca"
format_stamp: "Formatted at 2026-09-29 16:31:15 on dist-test-slave-mlmz"
server_key: "2f9fadffaafc304c453eb25c7169c860"
server_key_iv: "6fa084006c8523f8b9cfc0baad7cf7ae"
server_key_version: "encryptionkey@0"
I20260929 16:31:15.160159 28547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:15.164316 28547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:15.164809 28547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:15.164943 28547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:15.165423 28547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:15.166185 28547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:15.166229 28547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:15.166263 28547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:15.166277 28547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:15.174604 28547 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:35959
I20260929 16:31:15.174628 28674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:35959 every 8 connection(s)
I20260929 16:31:15.175382 28547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:15.180977 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.176942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:39569 (local address 127.23.44.62:38271)
0929 16:31:15.177105 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:15.177110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:15.178130 (+ 1020us) server_negotiation.cc:407] Connection header received
0929 16:31:15.179484 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:15.179486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:15.179522 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:15.179594 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:15.180574 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:15.180577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:15.180579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:15.180589 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:15.180607 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:15.180614 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:15.180687 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:15.180843 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 16:31:15.180882 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:31:15.181752 28677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.177397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38271 (local address 127.23.44.2:39569)
0929 16:31:15.177983 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:15.178019 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:15.179253 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:15.179735 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:15.179742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:15.179820 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:15.180332 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:15.180333 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:15.180368 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:15.180435 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:15.180688 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:15.180694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:15.180698 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:31:15.180817 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:31:15.180983 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":101,"threads_started":1}
I20260929 16:31:15.182855 28675 heartbeater.cc:347] Connected to a master server at 127.23.44.62:38271
I20260929 16:31:15.183079 28675 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:15.183440 28675 heartbeater.cc:511] Master 127.23.44.62:38271 requested a full tablet report, sending...
I20260929 16:31:15.184285 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 28547
I20260929 16:31:15.184419 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:15.184440 28356 ts_manager.cc:194] Registered new tserver with Master: cf23551073614f3f8931300433c7d6ca (127.23.44.2:35959)
I20260929 16:31:15.184695 23728 external_mini_cluster.cc:1428] Setting key 05b587d580d61a666f1498765b43e24a
I20260929 16:31:15.186476 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38271
--builtin_ntp_servers=127.23.44.20:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:31:15.294533 28678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:15.294757 28678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:15.294802 28678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:15.299243 28678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:15.299294 28678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:15.299347 28678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:15.304258 28678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:15.305177 28678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:15.306552 28678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:15.316361 28683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:15.316352 28684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:15.316406 28678 server_base.cc:1034] running on GCE node
W20260929 16:31:15.316373 28686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:15.316978 28678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:15.317656 28678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:15.318843 28678 hybrid_clock.cc:648] HybridClock initialized: now 1790699475318823 us; error 32 us; skew 500 ppm
I20260929 16:31:15.320520 28678 webserver.cc:460] Webserver started at http://127.23.44.3:37671/ using document root <none> and password file <none>
I20260929 16:31:15.321062 28678 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:15.321154 28678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:15.321439 28678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:15.323328 28678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "0e5f6bdaba58494aae619b0755f7c4f3"
format_stamp: "Formatted at 2026-09-29 16:31:15 on dist-test-slave-mlmz"
server_key: "6c0bceeddad1ac26634e7f0ce20e2cd5"
server_key_iv: "a28c842aa5efe63ae68c33da3ce08735"
server_key_version: "encryptionkey@0"
I20260929 16:31:15.323855 28678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "0e5f6bdaba58494aae619b0755f7c4f3"
format_stamp: "Formatted at 2026-09-29 16:31:15 on dist-test-slave-mlmz"
server_key: "6c0bceeddad1ac26634e7f0ce20e2cd5"
server_key_iv: "a28c842aa5efe63ae68c33da3ce08735"
server_key_version: "encryptionkey@0"
I20260929 16:31:15.327203 28678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:31:15.329500 28692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:15.330610 28678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:15.330732 28678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "0e5f6bdaba58494aae619b0755f7c4f3"
format_stamp: "Formatted at 2026-09-29 16:31:15 on dist-test-slave-mlmz"
server_key: "6c0bceeddad1ac26634e7f0ce20e2cd5"
server_key_iv: "a28c842aa5efe63ae68c33da3ce08735"
server_key_version: "encryptionkey@0"
I20260929 16:31:15.330835 28678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:15.335101 28678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:15.335705 28678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:15.335883 28678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:15.336414 28678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:15.337242 28678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:15.337368 28678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:15.337406 28678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:15.337421 28678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:15.345976 28678 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:42115
I20260929 16:31:15.346004 28805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:42115 every 8 connection(s)
I20260929 16:31:15.346736 28678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:15.351943 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 28678
I20260929 16:31:15.352051 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:15.352200 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.348319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:59751 (local address 127.23.44.62:38271)
0929 16:31:15.348494 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:15.348499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:15.349452 (+ 953us) server_negotiation.cc:407] Connection header received
0929 16:31:15.350656 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:15.350659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:15.350693 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:15.350779 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:15.351713 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:15.351715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:15.351717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:15.351725 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:15.351740 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:15.351747 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:15.351812 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:15.351975 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 16:31:15.352047 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:31:15.352320 23728 external_mini_cluster.cc:1428] Setting key 4621e4c7f0fb860c49645526c82406ff
I20260929 16:31:15.353096 28808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.348693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38271 (local address 127.23.44.3:59751)
0929 16:31:15.349320 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:15.349357 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:15.350464 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:15.350927 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:15.350933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:15.351010 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:15.351493 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:15.351495 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:15.351529 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:15.351586 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:15.351826 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:15.351837 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:15.351843 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:31:15.352014 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 16:31:15.352237 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":100,"threads_started":1}
I20260929 16:31:15.354242 28806 heartbeater.cc:347] Connected to a master server at 127.23.44.62:38271
I20260929 16:31:15.354509 28806 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:15.354893 28806 heartbeater.cc:511] Master 127.23.44.62:38271 requested a full tablet report, sending...
I20260929 16:31:15.355829 28355 ts_manager.cc:194] Registered new tserver with Master: 0e5f6bdaba58494aae619b0755f7c4f3 (127.23.44.3:42115)
I20260929 16:31:15.366108 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:31:15.369854 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.367998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.23.44.62:38271)
0929 16:31:15.368137 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:15.368143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:15.368286 (+ 143us) server_negotiation.cc:407] Connection header received
0929 16:31:15.368395 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:15.368397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:15.368426 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:15.368495 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:15.369371 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:15.369376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:15.369378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:15.369391 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:15.369411 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:15.369421 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:15.369493 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:15.369669 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 16:31:15.369708 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:31:15.372840 28355 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:15.375710 28355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:15.386400 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.383226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:42115 (local address 127.0.0.1:58024)
0929 16:31:15.383554 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:15.383602 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:15.383690 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:15.384632 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:15.384640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:15.384710 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:15.385303 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:15.385305 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:15.385325 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:15.385388 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:15.386065 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:15.386074 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:15.386080 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:31:15.386202 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 16:31:15.386266 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260929 16:31:15.386420 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.383364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58024 (local address 127.23.44.3:42115)
0929 16:31:15.384196 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:15.384208 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:15.384233 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:31:15.384285 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:15.384290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:15.384472 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:15.385029 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:15.385889 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:15.385894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:15.385902 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:15.385911 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:15.385964 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:15.385988 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:15.386092 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:15.386221 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 16:31:15.386287 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":136,"threads_started":1}
I20260929 16:31:15.386984 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.384471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.23.44.1:43389)
0929 16:31:15.384774 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:15.384786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:15.384923 (+ 137us) server_negotiation.cc:407] Connection header received
0929 16:31:15.384984 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:15.384989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:15.385141 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:15.385255 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:15.386175 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:15.386180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:15.386189 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:15.386199 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:15.386236 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:15.386262 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:15.386350 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:15.386759 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 16:31:15.386853 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
I20260929 16:31:15.387274 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.384095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:43389 (local address 127.0.0.1:41440)
0929 16:31:15.384817 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:15.384830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:15.384910 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:15.385334 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:15.385337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:15.385351 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:15.385957 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:15.385962 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:15.385971 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:15.385995 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:15.386387 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:15.386390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:15.386393 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 16:31:15.387126 (+ 733us) client_negotiation.cc:240] Negotiation successful
0929 16:31:15.387164 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":91,"threads_started":1}
I20260929 16:31:15.387319 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.383446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:35959 (local address 127.0.0.1:51332)
0929 16:31:15.383892 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:15.383904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:15.383969 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:15.385348 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:15.385350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:15.385363 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:15.385833 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:15.385836 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:15.385848 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:15.385875 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:15.386244 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:15.386247 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:15.386250 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 16:31:15.386340 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:31:15.386378 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":102,"threads_started":1}
I20260929 16:31:15.386695 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.383948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.23.44.2:35959)
0929 16:31:15.384267 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:15.384281 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:15.384305 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:31:15.384370 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:15.384375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:15.384541 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:15.384679 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:15.386006 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:15.386014 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:15.386026 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:15.386040 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:15.386077 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:15.386106 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:15.386201 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:15.386460 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 16:31:15.386537 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":124,"threads_started":1}
I20260929 16:31:15.389454 28740 tablet_service.cc:1467] Processing CreateTablet for tablet 683696bdc11c4b0caaada1a84e2685df (DEFAULT_TABLE table=test-table [id=602d0c80fa5144dcbca850a82dc90cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:15.389730 28478 tablet_service.cc:1467] Processing CreateTablet for tablet 683696bdc11c4b0caaada1a84e2685df (DEFAULT_TABLE table=test-table [id=602d0c80fa5144dcbca850a82dc90cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:15.390439 28740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683696bdc11c4b0caaada1a84e2685df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:15.390653 28478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683696bdc11c4b0caaada1a84e2685df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:15.390717 28609 tablet_service.cc:1467] Processing CreateTablet for tablet 683696bdc11c4b0caaada1a84e2685df (DEFAULT_TABLE table=test-table [id=602d0c80fa5144dcbca850a82dc90cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:15.391567 28609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683696bdc11c4b0caaada1a84e2685df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:15.396044 28823 tablet_bootstrap.cc:492] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3: Bootstrap starting.
I20260929 16:31:15.397727 28825 tablet_bootstrap.cc:492] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca: Bootstrap starting.
I20260929 16:31:15.398167 28824 tablet_bootstrap.cc:492] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4: Bootstrap starting.
I20260929 16:31:15.398597 28823 tablet_bootstrap.cc:654] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:15.399490 28823 log.cc:824] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:15.399802 28825 tablet_bootstrap.cc:654] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:15.400202 28824 tablet_bootstrap.cc:654] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:15.400746 28825 log.cc:824] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:15.400998 28824 log.cc:824] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:15.401732 28823 tablet_bootstrap.cc:492] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3: No bootstrap required, opened a new log
I20260929 16:31:15.401971 28823 ts_tablet_manager.cc:1397] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:31:15.402639 28825 tablet_bootstrap.cc:492] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca: No bootstrap required, opened a new log
I20260929 16:31:15.402813 28825 ts_tablet_manager.cc:1397] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:31:15.402936 28824 tablet_bootstrap.cc:492] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4: No bootstrap required, opened a new log
I20260929 16:31:15.403116 28824 ts_tablet_manager.cc:1397] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 16:31:15.405364 28823 raft_consensus.cc:348] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5f6bdaba58494aae619b0755f7c4f3" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42115 } } peers { permanent_uuid: "cf23551073614f3f8931300433c7d6ca" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35959 } } peers { permanent_uuid: "164e7a61b23a4e439af7c0bc9b1543d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43389 } }
I20260929 16:31:15.405601 28823 raft_consensus.cc:374] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:15.405658 28823 raft_consensus.cc:729] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5f6bdaba58494aae619b0755f7c4f3, State: Initialized, Role: FOLLOWER
I20260929 16:31:15.406085 28823 consensus_queue.cc:260] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5f6bdaba58494aae619b0755f7c4f3" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42115 } } peers { permanent_uuid: "cf23551073614f3f8931300433c7d6ca" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35959 } } peers { permanent_uuid: "164e7a61b23a4e439af7c0bc9b1543d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43389 } }
I20260929 16:31:15.406639 28806 heartbeater.cc:503] Master 127.23.44.62:38271 was elected leader, sending a full tablet report...
I20260929 16:31:15.406996 28823 ts_tablet_manager.cc:1428] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:31:15.407266 28824 raft_consensus.cc:348] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5f6bdaba58494aae619b0755f7c4f3" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42115 } } peers { permanent_uuid: "cf23551073614f3f8931300433c7d6ca" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35959 } } peers { permanent_uuid: "164e7a61b23a4e439af7c0bc9b1543d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43389 } }
I20260929 16:31:15.407244 28825 raft_consensus.cc:348] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5f6bdaba58494aae619b0755f7c4f3" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42115 } } peers { permanent_uuid: "cf23551073614f3f8931300433c7d6ca" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35959 } } peers { permanent_uuid: "164e7a61b23a4e439af7c0bc9b1543d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43389 } }
I20260929 16:31:15.407482 28825 raft_consensus.cc:374] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:15.407482 28824 raft_consensus.cc:374] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:15.407537 28824 raft_consensus.cc:729] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164e7a61b23a4e439af7c0bc9b1543d4, State: Initialized, Role: FOLLOWER
I20260929 16:31:15.407545 28825 raft_consensus.cc:729] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf23551073614f3f8931300433c7d6ca, State: Initialized, Role: FOLLOWER
I20260929 16:31:15.407927 28825 consensus_queue.cc:260] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5f6bdaba58494aae619b0755f7c4f3" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42115 } } peers { permanent_uuid: "cf23551073614f3f8931300433c7d6ca" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35959 } } peers { permanent_uuid: "164e7a61b23a4e439af7c0bc9b1543d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43389 } }
I20260929 16:31:15.407927 28824 consensus_queue.cc:260] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5f6bdaba58494aae619b0755f7c4f3" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42115 } } peers { permanent_uuid: "cf23551073614f3f8931300433c7d6ca" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35959 } } peers { permanent_uuid: "164e7a61b23a4e439af7c0bc9b1543d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43389 } }
I20260929 16:31:15.408787 28825 ts_tablet_manager.cc:1428] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:31:15.408924 28544 heartbeater.cc:503] Master 127.23.44.62:38271 was elected leader, sending a full tablet report...
I20260929 16:31:15.409003 28675 heartbeater.cc:503] Master 127.23.44.62:38271 was elected leader, sending a full tablet report...
I20260929 16:31:15.408787 28824 ts_tablet_manager.cc:1428] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
W20260929 16:31:15.426222 28676 tablet.cc:2369] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:31:15.426223 28831 raft_consensus.cc:482] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:15.426434 28831 raft_consensus.cc:504] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5f6bdaba58494aae619b0755f7c4f3" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42115 } } peers { permanent_uuid: "cf23551073614f3f8931300433c7d6ca" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35959 } } peers { permanent_uuid: "164e7a61b23a4e439af7c0bc9b1543d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43389 } }
I20260929 16:31:15.427508 28831 leader_election.cc:288] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e5f6bdaba58494aae619b0755f7c4f3 (127.23.44.3:42115), 164e7a61b23a4e439af7c0bc9b1543d4 (127.23.44.1:43389)
I20260929 16:31:15.429450 28677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.427688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:42115 (local address 127.23.44.2:33067)
0929 16:31:15.427824 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:15.427840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:15.427913 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:15.428239 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:15.428242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:15.428255 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:15.428784 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:15.428789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:15.428801 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:15.428828 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:15.429135 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:15.429139 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:15.429143 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:31:15.429251 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:31:15.429311 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:31:15.429665 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.427799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:33067 (local address 127.23.44.3:42115)
0929 16:31:15.427937 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:15.427942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:15.427958 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:31:15.428001 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:15.428006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:15.428040 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:15.428145 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:15.428964 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:15.428971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:15.428974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:15.428989 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:15.429014 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:15.429026 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:15.429119 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:15.429471 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 16:31:15.429534 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:31:15.429729 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.427967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:43389 (local address 127.23.44.2:48299)
0929 16:31:15.428269 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:15.428281 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:15.428367 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:15.428601 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:15.428603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:15.428635 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:15.429135 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:15.429137 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:15.429147 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:15.429174 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:15.429446 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:15.429449 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:15.429452 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 16:31:15.429541 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 16:31:15.429576 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260929 16:31:15.429854 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.428087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:48299 (local address 127.23.44.1:43389)
0929 16:31:15.428195 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:15.428202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:15.428353 (+ 151us) server_negotiation.cc:407] Connection header received
0929 16:31:15.428459 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:15.428464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:15.428497 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:15.428576 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:15.429319 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:15.429322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:15.429325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:15.429343 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:15.429359 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:15.429366 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:15.429436 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:15.429701 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 16:31:15.429740 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 16:31:15.430234 28760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683696bdc11c4b0caaada1a84e2685df" candidate_uuid: "cf23551073614f3f8931300433c7d6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5f6bdaba58494aae619b0755f7c4f3" is_pre_election: true
I20260929 16:31:15.430456 28498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683696bdc11c4b0caaada1a84e2685df" candidate_uuid: "cf23551073614f3f8931300433c7d6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "164e7a61b23a4e439af7c0bc9b1543d4" is_pre_election: true
I20260929 16:31:15.430507 28760 raft_consensus.cc:2454] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf23551073614f3f8931300433c7d6ca in term 0.
I20260929 16:31:15.430697 28498 raft_consensus.cc:2454] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf23551073614f3f8931300433c7d6ca in term 0.
I20260929 16:31:15.431003 28562 leader_election.cc:302] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5f6bdaba58494aae619b0755f7c4f3, cf23551073614f3f8931300433c7d6ca; no voters:
I20260929 16:31:15.431272 28831 raft_consensus.cc:2781] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:15.431353 28831 raft_consensus.cc:482] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:15.431397 28831 raft_consensus.cc:3037] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:15.432367 28831 raft_consensus.cc:504] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5f6bdaba58494aae619b0755f7c4f3" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42115 } } peers { permanent_uuid: "cf23551073614f3f8931300433c7d6ca" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35959 } } peers { permanent_uuid: "164e7a61b23a4e439af7c0bc9b1543d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43389 } }
I20260929 16:31:15.432667 28831 leader_election.cc:288] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [CANDIDATE]: Term 1 election: Requested vote from peers 0e5f6bdaba58494aae619b0755f7c4f3 (127.23.44.3:42115), 164e7a61b23a4e439af7c0bc9b1543d4 (127.23.44.1:43389)
I20260929 16:31:15.432999 28760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683696bdc11c4b0caaada1a84e2685df" candidate_uuid: "cf23551073614f3f8931300433c7d6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5f6bdaba58494aae619b0755f7c4f3"
I20260929 16:31:15.433130 28760 raft_consensus.cc:3037] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:15.433139 28498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683696bdc11c4b0caaada1a84e2685df" candidate_uuid: "cf23551073614f3f8931300433c7d6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "164e7a61b23a4e439af7c0bc9b1543d4"
I20260929 16:31:15.433243 28498 raft_consensus.cc:3037] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:15.434322 28760 raft_consensus.cc:2454] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf23551073614f3f8931300433c7d6ca in term 1.
I20260929 16:31:15.434348 28498 raft_consensus.cc:2454] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf23551073614f3f8931300433c7d6ca in term 1.
I20260929 16:31:15.434643 28562 leader_election.cc:302] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5f6bdaba58494aae619b0755f7c4f3, cf23551073614f3f8931300433c7d6ca; no voters:
I20260929 16:31:15.434823 28831 raft_consensus.cc:2781] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:15.435026 28831 raft_consensus.cc:686] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 1 LEADER]: Becoming Leader. State: Replica: cf23551073614f3f8931300433c7d6ca, State: Running, Role: LEADER
I20260929 16:31:15.435314 28831 consensus_queue.cc:237] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5f6bdaba58494aae619b0755f7c4f3" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42115 } } peers { permanent_uuid: "cf23551073614f3f8931300433c7d6ca" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35959 } } peers { permanent_uuid: "164e7a61b23a4e439af7c0bc9b1543d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43389 } }
I20260929 16:31:15.437979 28354 catalog_manager.cc:5495] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca reported cstate change: term changed from 0 to 1, leader changed from <none> to cf23551073614f3f8931300433c7d6ca (127.23.44.2). New cstate: current_term: 1 leader_uuid: "cf23551073614f3f8931300433c7d6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5f6bdaba58494aae619b0755f7c4f3" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf23551073614f3f8931300433c7d6ca" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164e7a61b23a4e439af7c0bc9b1543d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43389 } health_report { overall_health: UNKNOWN } } }
I20260929 16:31:15.451326 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.450084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.23.44.2:35959)
0929 16:31:15.450197 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:15.450202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:15.450222 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:31:15.450324 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:15.450326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:15.450382 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:15.450448 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:15.451047 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:15.451051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:15.451053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:15.451063 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:15.451080 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:15.451087 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:15.451164 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:15.451205 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 16:31:15.451233 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:31:15.456262 28498 raft_consensus.cc:1261] T 683696bdc11c4b0caaada1a84e2685df P 164e7a61b23a4e439af7c0bc9b1543d4 [term 1 FOLLOWER]: Refusing update from remote peer cf23551073614f3f8931300433c7d6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:15.456476 28760 raft_consensus.cc:1261] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3 [term 1 FOLLOWER]: Refusing update from remote peer cf23551073614f3f8931300433c7d6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:15.456928 28831 consensus_queue.cc:1035] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "164e7a61b23a4e439af7c0bc9b1543d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:15.457113 28833 consensus_queue.cc:1035] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5f6bdaba58494aae619b0755f7c4f3" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:15.465622 28837 mvcc.cc:205] Tried to move back new op lower bound from 7334705051462959104 to 7334705051385282560. Current Snapshot: MvccSnapshot[applied={T|T < 7334705051462959104}]
I20260929 16:31:15.465760 28838 mvcc.cc:205] Tried to move back new op lower bound from 7334705051462959104 to 7334705051385282560. Current Snapshot: MvccSnapshot[applied={T|T < 7334705051462959104}]
I20260929 16:31:15.471694 28354 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:15.471858 28354 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:15.474236 28354 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43bea382fece45439fd8a06fbda01aac: Sending DeleteTablet for 3 replicas of tablet 683696bdc11c4b0caaada1a84e2685df
I20260929 16:31:15.475010 28609 tablet_service.cc:1514] Processing DeleteTablet for tablet 683696bdc11c4b0caaada1a84e2685df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:15 UTC) from {username='slave'} at 127.0.0.1:51332
I20260929 16:31:15.475059 28740 tablet_service.cc:1514] Processing DeleteTablet for tablet 683696bdc11c4b0caaada1a84e2685df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:15 UTC) from {username='slave'} at 127.0.0.1:58024
I20260929 16:31:15.475190 28478 tablet_service.cc:1514] Processing DeleteTablet for tablet 683696bdc11c4b0caaada1a84e2685df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:15 UTC) from {username='slave'} at 127.0.0.1:41440
I20260929 16:31:15.475500 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 28416
I20260929 16:31:15.475579 28846 tablet_replica.cc:330] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3: stopping tablet replica
I20260929 16:31:15.476147 28846 raft_consensus.cc:2229] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:15.476434 28846 raft_consensus.cc:2258] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:15.476480 28845 tablet_replica.cc:330] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca: stopping tablet replica
I20260929 16:31:15.477035 28845 raft_consensus.cc:2229] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:15.477465 28845 raft_consensus.cc:2258] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:15.477914 28846 ts_tablet_manager.cc:1905] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:15.480571 28846 ts_tablet_manager.cc:1918] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:15.480669 28846 log.cc:1196] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/683696bdc11c4b0caaada1a84e2685df
I20260929 16:31:15.480993 28846 ts_tablet_manager.cc:1939] T 683696bdc11c4b0caaada1a84e2685df P 0e5f6bdaba58494aae619b0755f7c4f3: Deleting consensus metadata
I20260929 16:31:15.481870 28845 ts_tablet_manager.cc:1905] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:15.482192 28340 catalog_manager.cc:4841] TS 0e5f6bdaba58494aae619b0755f7c4f3 (127.23.44.3:42115): tablet 683696bdc11c4b0caaada1a84e2685df (table test-table [id=602d0c80fa5144dcbca850a82dc90cce]) successfully deleted
I20260929 16:31:15.485313 28845 ts_tablet_manager.cc:1918] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:15.485412 28845 log.cc:1196] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/683696bdc11c4b0caaada1a84e2685df
I20260929 16:31:15.485761 28845 ts_tablet_manager.cc:1939] T 683696bdc11c4b0caaada1a84e2685df P cf23551073614f3f8931300433c7d6ca: Deleting consensus metadata
W20260929 16:31:15.486064 28340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:43389 (error 108)
I20260929 16:31:15.486830 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 28547
I20260929 16:31:15.487049 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.486736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:43389 (local address 127.0.0.1:41442)
0929 16:31:15.486917 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:15.486978 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260929 16:31:15.487602 28340 catalog_manager.cc:4584] TS 164e7a61b23a4e439af7c0bc9b1543d4 (127.23.44.1:43389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 683696bdc11c4b0caaada1a84e2685df: Network error: Client connection negotiation failed: client connection to 127.23.44.1:43389: connect: Connection refused (error 111)
I20260929 16:31:15.492354 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 28678
I20260929 16:31:15.493983 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.492427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:35959 (local address 127.0.0.1:51342)
0929 16:31:15.493817 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:15.493881 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.2:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1288}
I20260929 16:31:15.497648 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 28325
2026-09-29T16:31:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (904 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 16:31:15.518362 23728 mini_oidc.cc:152] Starting JWKS server
I20260929 16:31:15.519853 23728 webserver.cc:460] Webserver started at https://127.0.0.1:35455/ using document root <none> and password file <none>
I20260929 16:31:15.520032 23728 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:31:15.520299 23728 webserver.cc:460] Webserver started at http://127.0.0.1:41187/ using document root <none> and password file <none>
2026-09-29T16:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:15Z Disabled control of system clock
I20260929 16:31:15.528630 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:45075
--webserver_interface=127.23.44.62
--webserver_port=0
--builtin_ntp_servers=127.23.44.20:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:45075
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35455/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/testchainca.cert with env {}
W20260929 16:31:15.632413 28854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:15.632682 28854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:15.632752 28854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:15.636704 28854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:15.636790 28854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:15.636819 28854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:15.636904 28854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:15.636945 28854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:31:15.636978 28854 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:31:15.637003 28854 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35455/jwks.json
I20260929 16:31:15.642079 28854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:45075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35455/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:15.643014 28854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:15.643934 28854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:15.651889 28862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:15.651938 28859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:15.651916 28860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:15.652132 28854 server_base.cc:1034] running on GCE node
I20260929 16:31:15.652807 28854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:15.653862 28854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:15.655061 28854 hybrid_clock.cc:648] HybridClock initialized: now 1790699475655049 us; error 25 us; skew 500 ppm
I20260929 16:31:15.656841 28854 webserver.cc:460] Webserver started at http://127.23.44.62:38691/ using document root <none> and password file <none>
I20260929 16:31:15.657454 28854 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:15.657541 28854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:15.657756 28854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:15.659740 28854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "e532ef050d1644edaaf5955a097ba2f9"
format_stamp: "Formatted at 2026-09-29 16:31:15 on dist-test-slave-mlmz"
server_key: "edb2dea089a9f6f6b57f87f51d9ba685"
server_key_iv: "244f7644ea2c47eb22ee88e0bcc46327"
server_key_version: "encryptionkey@0"
I20260929 16:31:15.660373 28854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "e532ef050d1644edaaf5955a097ba2f9"
format_stamp: "Formatted at 2026-09-29 16:31:15 on dist-test-slave-mlmz"
server_key: "edb2dea089a9f6f6b57f87f51d9ba685"
server_key_iv: "244f7644ea2c47eb22ee88e0bcc46327"
server_key_version: "encryptionkey@0"
I20260929 16:31:15.663846 28854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 16:31:15.666196 28868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:15.667433 28854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:15.667565 28854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "e532ef050d1644edaaf5955a097ba2f9"
format_stamp: "Formatted at 2026-09-29 16:31:15 on dist-test-slave-mlmz"
server_key: "edb2dea089a9f6f6b57f87f51d9ba685"
server_key_iv: "244f7644ea2c47eb22ee88e0bcc46327"
server_key_version: "encryptionkey@0"
I20260929 16:31:15.667666 28854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:15.703506 28854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:15.704172 28854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:15.704365 28854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:15.711277 28854 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:45075
I20260929 16:31:15.711297 28920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:45075 every 8 connection(s)
I20260929 16:31:15.712191 28854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:15.713824 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 28854
I20260929 16:31:15.713948 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:15.714267 23728 external_mini_cluster.cc:1428] Setting key c798f48aa383dcdc9f55addf37b18caf
I20260929 16:31:15.716325 28921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:15.722263 28921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9: Bootstrap starting.
I20260929 16:31:15.723248 28924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.715678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.23.44.62:45075)
0929 16:31:15.716307 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:15.716322 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:15.716358 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:31:15.717029 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:15.717044 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:15.717490 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:15.717814 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:15.718223 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:15.719167 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:15.720046 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:15.720522 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:15.720690 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:15.721650 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:15.721655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:15.721666 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:15.722037 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:15.722088 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:15.722108 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:15.722216 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:15.722325 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 16:31:15.722587 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":145,"threads_started":1}
I20260929 16:31:15.724711 28921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:15.725709 28921 log.cc:824] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:15.726300 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:31:15.727963 28921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9: No bootstrap required, opened a new log
I20260929 16:31:15.732079 28921 raft_consensus.cc:348] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e532ef050d1644edaaf5955a097ba2f9" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45075 } }
I20260929 16:31:15.732368 28921 raft_consensus.cc:374] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:15.732411 28921 raft_consensus.cc:729] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e532ef050d1644edaaf5955a097ba2f9, State: Initialized, Role: FOLLOWER
I20260929 16:31:15.732780 28921 consensus_queue.cc:260] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e532ef050d1644edaaf5955a097ba2f9" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45075 } }
I20260929 16:31:15.732898 28921 raft_consensus.cc:388] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:15.732945 28921 raft_consensus.cc:482] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:15.733018 28921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:15.734239 28921 raft_consensus.cc:504] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e532ef050d1644edaaf5955a097ba2f9" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45075 } }
I20260929 16:31:15.734547 28921 leader_election.cc:302] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e532ef050d1644edaaf5955a097ba2f9; no voters:
I20260929 16:31:15.734845 28921 leader_election.cc:288] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:15.735042 28926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:15.735321 28926 raft_consensus.cc:686] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [term 1 LEADER]: Becoming Leader. State: Replica: e532ef050d1644edaaf5955a097ba2f9, State: Running, Role: LEADER
I20260929 16:31:15.735631 28926 consensus_queue.cc:237] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e532ef050d1644edaaf5955a097ba2f9" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45075 } }
I20260929 16:31:15.736130 28921 sys_catalog.cc:564] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:15.737323 28928 sys_catalog.cc:455] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e532ef050d1644edaaf5955a097ba2f9. Latest consensus state: current_term: 1 leader_uuid: "e532ef050d1644edaaf5955a097ba2f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e532ef050d1644edaaf5955a097ba2f9" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45075 } } }
I20260929 16:31:15.737438 28928 sys_catalog.cc:458] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:15.737705 28927 sys_catalog.cc:455] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e532ef050d1644edaaf5955a097ba2f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e532ef050d1644edaaf5955a097ba2f9" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45075 } } }
I20260929 16:31:15.737794 28927 sys_catalog.cc:458] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:15.737845 28935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:15.741519 28935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:15.746923 28935 catalog_manager.cc:1347] Generated new cluster ID: 305afa7051cb4375bfaa9cd37b600107
I20260929 16:31:15.747004 28935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:15.766024 28935 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:15.766888 28935 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:15.771425 28935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e532ef050d1644edaaf5955a097ba2f9: Generated new TSK 0
I20260929 16:31:15.772169 28935 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 16:31:15.802137 28951 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 16:31:15.802258 28951 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:31:15.802759 28924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:15.786127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.23.44.62:45075)
0929 16:31:15.786263 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:15.786268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:15.786461 (+ 193us) server_negotiation.cc:407] Connection header received
0929 16:31:15.786672 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:15.786674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:15.786725 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:15.786816 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:31:15.787126 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:15.787631 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:15.788850 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:15.789059 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:15.789141 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:15.789525 (+ 384us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:31:15.802445 (+ 12920us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35455/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 16:31:15.802652 (+ 207us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46038: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35455/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 16:31:15.802839 28924 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46038: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35455/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 16:31:15.804165 28950 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:15.786045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:45075 (local address 127.0.0.1:46038)
0929 16:31:15.786375 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:15.786389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:15.786512 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:15.786855 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:15.786858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:15.786866 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:31:15.786983 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:15.786990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:15.787738 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:15.787741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:15.788720 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:15.788730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:15.789174 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:15.789178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:15.789293 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:15.789352 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:31:15.802656 (+ 13304us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35455/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 16:31:15.802925 (+ 269us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.44.62:45075: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35455/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":119,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.44.62:45075: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35455/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 16:31:15.805083 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 28854
2026-09-29T16:31:15Z chronyd exiting
I20260929 16:31:15.842155 23728 test_util.cc:175] -----------------------------------------------
I20260929 16:31:15.842234 23728 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699450427356-23728-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (325 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 16:31:15.843204 23728 mini_oidc.cc:152] Starting JWKS server
I20260929 16:31:15.844048 23728 webserver.cc:460] Webserver started at https://127.0.0.1:42387/ using document root <none> and password file <none>
I20260929 16:31:15.844153 23728 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:31:15.844420 23728 webserver.cc:460] Webserver started at http://127.0.0.1:40431/ using document root <none> and password file <none>
2026-09-29T16:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:15Z Disabled control of system clock
I20260929 16:31:15.852964 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:42555
--webserver_interface=127.23.44.62
--webserver_port=0
--builtin_ntp_servers=127.23.44.20:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:42555
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42387/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/testchainca.cert with env {}
W20260929 16:31:15.955422 28957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:15.955698 28957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:15.955770 28957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:15.959887 28957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:15.959991 28957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:15.960031 28957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:15.960069 28957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:15.960098 28957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:31:15.960135 28957 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:31:15.960166 28957 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42387/jwks.json
I20260929 16:31:15.965378 28957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:42555
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:42555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42387/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:15.966339 28957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:15.967288 28957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:15.975452 28962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:15.975452 28965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:15.975471 28963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:15.975488 28957 server_base.cc:1034] running on GCE node
I20260929 16:31:15.976176 28957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:15.977219 28957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:15.978456 28957 hybrid_clock.cc:648] HybridClock initialized: now 1790699475978422 us; error 44 us; skew 500 ppm
I20260929 16:31:15.980265 28957 webserver.cc:460] Webserver started at http://127.23.44.62:34637/ using document root <none> and password file <none>
I20260929 16:31:15.980880 28957 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:15.980978 28957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:15.981215 28957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:15.983424 28957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "abcc382cb37248c7a87221931245970b"
format_stamp: "Formatted at 2026-09-29 16:31:15 on dist-test-slave-mlmz"
server_key: "d9f44821d5d2c0fa977c66eb1cc66666"
server_key_iv: "3667f57643f7c0bc38900c69d3270013"
server_key_version: "encryptionkey@0"
I20260929 16:31:15.984006 28957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "abcc382cb37248c7a87221931245970b"
format_stamp: "Formatted at 2026-09-29 16:31:15 on dist-test-slave-mlmz"
server_key: "d9f44821d5d2c0fa977c66eb1cc66666"
server_key_iv: "3667f57643f7c0bc38900c69d3270013"
server_key_version: "encryptionkey@0"
I20260929 16:31:15.987552 28957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260929 16:31:15.989944 28971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:15.991158 28957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:15.991303 28957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "abcc382cb37248c7a87221931245970b"
format_stamp: "Formatted at 2026-09-29 16:31:15 on dist-test-slave-mlmz"
server_key: "d9f44821d5d2c0fa977c66eb1cc66666"
server_key_iv: "3667f57643f7c0bc38900c69d3270013"
server_key_version: "encryptionkey@0"
I20260929 16:31:15.991418 28957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:16.008292 28957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:16.008924 28957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:16.009115 28957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:16.016136 28957 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:42555
I20260929 16:31:16.016148 29023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:42555 every 8 connection(s)
I20260929 16:31:16.017036 28957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:16.017706 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 28957
I20260929 16:31:16.017835 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:16.018097 23728 external_mini_cluster.cc:1428] Setting key f3de620bfff8ead0bd564cc136ec4c4c
I20260929 16:31:16.020432 29024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:16.025641 29024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b: Bootstrap starting.
I20260929 16:31:16.026889 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:16.019348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.23.44.62:42555)
0929 16:31:16.020013 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:16.020035 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:16.020080 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:31:16.020777 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:16.020794 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:16.021172 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:16.021554 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:16.021953 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.022885 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.023765 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.024213 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.024380 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:16.025240 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:16.025245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:16.025297 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:16.025687 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:16.025737 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:16.025781 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:16.025872 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:16.026004 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 16:31:16.026242 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":138,"threads_started":1}
I20260929 16:31:16.027951 29024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:16.028714 29024 log.cc:824] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:16.029716 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:31:16.030701 29024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b: No bootstrap required, opened a new log
I20260929 16:31:16.034294 29024 raft_consensus.cc:348] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcc382cb37248c7a87221931245970b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42555 } }
I20260929 16:31:16.034493 29024 raft_consensus.cc:374] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:16.034554 29024 raft_consensus.cc:729] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abcc382cb37248c7a87221931245970b, State: Initialized, Role: FOLLOWER
I20260929 16:31:16.035017 29024 consensus_queue.cc:260] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcc382cb37248c7a87221931245970b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42555 } }
I20260929 16:31:16.035132 29024 raft_consensus.cc:388] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:16.035215 29024 raft_consensus.cc:482] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:16.035329 29024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:16.036191 29024 raft_consensus.cc:504] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcc382cb37248c7a87221931245970b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42555 } }
I20260929 16:31:16.036518 29024 leader_election.cc:302] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abcc382cb37248c7a87221931245970b; no voters:
I20260929 16:31:16.036819 29024 leader_election.cc:288] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:16.036962 29029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:16.037192 29029 raft_consensus.cc:686] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [term 1 LEADER]: Becoming Leader. State: Replica: abcc382cb37248c7a87221931245970b, State: Running, Role: LEADER
I20260929 16:31:16.037722 29029 consensus_queue.cc:237] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcc382cb37248c7a87221931245970b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42555 } }
I20260929 16:31:16.038254 29024 sys_catalog.cc:564] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:16.039366 29030 sys_catalog.cc:455] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abcc382cb37248c7a87221931245970b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcc382cb37248c7a87221931245970b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42555 } } }
I20260929 16:31:16.039386 29031 sys_catalog.cc:455] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [sys.catalog]: SysCatalogTable state changed. Reason: New leader abcc382cb37248c7a87221931245970b. Latest consensus state: current_term: 1 leader_uuid: "abcc382cb37248c7a87221931245970b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcc382cb37248c7a87221931245970b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42555 } } }
I20260929 16:31:16.039507 29030 sys_catalog.cc:458] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:16.039507 29031 sys_catalog.cc:458] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:16.039971 29038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:16.042699 29038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:16.047820 29038 catalog_manager.cc:1347] Generated new cluster ID: c8d6b184d43f48d2aa1718e105e015fa
I20260929 16:31:16.047880 29038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:16.065595 29038 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:16.066471 29038 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:16.075905 29038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abcc382cb37248c7a87221931245970b: Generated new TSK 0
I20260929 16:31:16.076563 29038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:31:16.112118 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:16.088992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.23.44.62:42555)
0929 16:31:16.089122 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:16.089126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:16.089323 (+ 197us) server_negotiation.cc:407] Connection header received
0929 16:31:16.089486 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:16.089489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:16.089532 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:16.089614 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:31:16.089847 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.090357 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.091570 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.091769 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.091846 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:16.092143 (+ 297us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:31:16.111715 (+ 19572us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 16:31:16.111972 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40502: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 16:31:16.112192 29027 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40502: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 16:31:16.112290 29053 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:16.088922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42555 (local address 127.0.0.1:40502)
0929 16:31:16.089209 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:16.089224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:16.089368 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:16.089638 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:16.089641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:16.089649 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:31:16.089758 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:16.089764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.090464 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.090467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:16.091457 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:16.091466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.091883 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.091887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:16.091971 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:16.091997 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:31:16.111935 (+ 19938us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 16:31:16.112093 (+ 158us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.44.62:42555: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":114,"threads_started":1}
E20260929 16:31:16.112396 29054 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 16:31:16.112453 29054 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:31:16.113065 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 28957
2026-09-29T16:31:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (307 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 16:31:16.151101 23728 mini_oidc.cc:152] Starting JWKS server
I20260929 16:31:16.151896 23728 webserver.cc:460] Webserver started at https://127.0.0.1:46169/ using document root <none> and password file <none>
I20260929 16:31:16.151971 23728 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:31:16.152208 23728 webserver.cc:460] Webserver started at http://127.0.0.1:38493/ using document root <none> and password file <none>
2026-09-29T16:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:16Z Disabled control of system clock
I20260929 16:31:16.160403 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:35761
--webserver_interface=127.23.44.62
--webserver_port=0
--builtin_ntp_servers=127.23.44.20:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:35761
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46169/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 16:31:16.264798 29060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:16.265023 29060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:16.265059 29060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:16.269603 29060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:16.269658 29060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:16.269673 29060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:16.269690 29060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:16.269704 29060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:31:16.269719 29060 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:31:16.269732 29060 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46169/jwks.json
I20260929 16:31:16.274295 29060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:35761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:35761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46169/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:16.275254 29060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:16.276473 29060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:16.284512 29068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:16.284662 29060 server_base.cc:1034] running on GCE node
W20260929 16:31:16.284562 29065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:16.284525 29066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:16.285346 29060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:16.286345 29060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:16.287539 29060 hybrid_clock.cc:648] HybridClock initialized: now 1790699476287522 us; error 29 us; skew 500 ppm
I20260929 16:31:16.289218 29060 webserver.cc:460] Webserver started at http://127.23.44.62:42417/ using document root <none> and password file <none>
I20260929 16:31:16.289794 29060 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:16.289882 29060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:16.290099 29060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:16.292014 29060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "cbf399ad2d9a4132b206a1fd01e9aa03"
format_stamp: "Formatted at 2026-09-29 16:31:16 on dist-test-slave-mlmz"
server_key: "21aa8ca14c5c9bb6de1e8f4ea0139d96"
server_key_iv: "459b78d41697919fdea6d6ad379b8c87"
server_key_version: "encryptionkey@0"
I20260929 16:31:16.292557 29060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "cbf399ad2d9a4132b206a1fd01e9aa03"
format_stamp: "Formatted at 2026-09-29 16:31:16 on dist-test-slave-mlmz"
server_key: "21aa8ca14c5c9bb6de1e8f4ea0139d96"
server_key_iv: "459b78d41697919fdea6d6ad379b8c87"
server_key_version: "encryptionkey@0"
I20260929 16:31:16.295920 29060 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 16:31:16.298166 29074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:16.299288 29060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:16.299441 29060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "cbf399ad2d9a4132b206a1fd01e9aa03"
format_stamp: "Formatted at 2026-09-29 16:31:16 on dist-test-slave-mlmz"
server_key: "21aa8ca14c5c9bb6de1e8f4ea0139d96"
server_key_iv: "459b78d41697919fdea6d6ad379b8c87"
server_key_version: "encryptionkey@0"
I20260929 16:31:16.299548 29060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:16.316515 29060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:16.317185 29060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:16.317399 29060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:16.323891 29060 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:35761
I20260929 16:31:16.323915 29126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:35761 every 8 connection(s)
I20260929 16:31:16.324754 29060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:16.325479 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 29060
I20260929 16:31:16.325595 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:16.325881 23728 external_mini_cluster.cc:1428] Setting key 0b80a68b6676b19cf434a5648a39b7bc
I20260929 16:31:16.327988 29127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:16.333135 29127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03: Bootstrap starting.
I20260929 16:31:16.334547 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:16.327169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.23.44.62:35761)
0929 16:31:16.327768 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:16.327785 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:16.327818 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:31:16.328499 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:16.328516 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:16.328870 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:16.329200 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:16.329627 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.330556 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.331413 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.331864 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.332032 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:16.332892 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:16.332897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:16.332908 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:16.333348 (+ 440us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:16.333401 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:16.333420 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:16.333532 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:16.333672 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 16:31:16.333907 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":145,"threads_started":1}
I20260929 16:31:16.335325 29127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:16.336150 29127 log.cc:824] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:16.337337 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:31:16.338194 29127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03: No bootstrap required, opened a new log
I20260929 16:31:16.341570 29127 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf399ad2d9a4132b206a1fd01e9aa03" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35761 } }
I20260929 16:31:16.341790 29127 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:16.341866 29127 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf399ad2d9a4132b206a1fd01e9aa03, State: Initialized, Role: FOLLOWER
I20260929 16:31:16.342303 29127 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf399ad2d9a4132b206a1fd01e9aa03" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35761 } }
I20260929 16:31:16.342453 29127 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:16.342532 29127 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:16.342624 29127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:16.343673 29127 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf399ad2d9a4132b206a1fd01e9aa03" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35761 } }
I20260929 16:31:16.343976 29127 leader_election.cc:302] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf399ad2d9a4132b206a1fd01e9aa03; no voters:
I20260929 16:31:16.344201 29127 leader_election.cc:288] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:16.344342 29132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:16.344594 29132 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [term 1 LEADER]: Becoming Leader. State: Replica: cbf399ad2d9a4132b206a1fd01e9aa03, State: Running, Role: LEADER
I20260929 16:31:16.344875 29132 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf399ad2d9a4132b206a1fd01e9aa03" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35761 } }
I20260929 16:31:16.345404 29127 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:16.346418 29133 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbf399ad2d9a4132b206a1fd01e9aa03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf399ad2d9a4132b206a1fd01e9aa03" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35761 } } }
I20260929 16:31:16.346540 29133 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:16.346953 29141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:16.346953 29134 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf399ad2d9a4132b206a1fd01e9aa03. Latest consensus state: current_term: 1 leader_uuid: "cbf399ad2d9a4132b206a1fd01e9aa03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf399ad2d9a4132b206a1fd01e9aa03" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35761 } } }
I20260929 16:31:16.347160 29134 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:16.349972 29141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:16.355399 29141 catalog_manager.cc:1347] Generated new cluster ID: 6133776cd52540cc98359b47c697f10c
I20260929 16:31:16.355485 29141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:16.364506 29141 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:16.365403 29141 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:16.371126 29141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbf399ad2d9a4132b206a1fd01e9aa03: Generated new TSK 0
I20260929 16:31:16.371727 29141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:31:16.392956 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:16.387846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.23.44.62:35761)
0929 16:31:16.387996 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:16.388001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:16.388135 (+ 134us) server_negotiation.cc:407] Connection header received
0929 16:31:16.388350 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:16.388353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:16.388411 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:16.388551 (+ 140us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:31:16.388796 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.389461 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.390419 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.390674 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.390796 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:16.391124 (+ 328us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:31:16.392574 (+ 1450us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 16:31:16.392805 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56924: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 16:31:16.393026 29130 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56924: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 16:31:16.395013 29156 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:16.387736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:35761 (local address 127.0.0.1:56924)
0929 16:31:16.388030 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:16.388044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:16.388189 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:16.388533 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:16.388537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:16.388545 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:31:16.388669 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:16.388677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.389583 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.389588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:16.390277 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:16.390285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.390803 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.390807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:16.390905 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:16.390939 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:31:16.392739 (+ 1800us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 16:31:16.394834 (+ 2095us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.44.62:35761: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1912,"thread_start_us":101,"threads_started":1}
I20260929 16:31:16.395313 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 29060
E20260929 16:31:16.404822 29157 webserver.cc:533] Webserver: sslize failed:
E20260929 16:31:16.404922 29157 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T16:31:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (283 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T16:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:16Z Disabled control of system clock
I20260929 16:31:16.442577 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:35719
--webserver_interface=127.23.44.62
--webserver_port=0
--builtin_ntp_servers=127.23.44.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:35719
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/testchainca.cert with env {}
W20260929 16:31:16.548019 29161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:16.548419 29161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:16.548501 29161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:16.554147 29161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:16.554200 29161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:16.554216 29161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:16.554232 29161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:16.554268 29161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:31:16.554298 29161 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:31:16.554327 29161 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 16:31:16.559597 29161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:35719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:35719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:16.560595 29161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:16.561805 29161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:16.570089 29166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:16.570166 29161 server_base.cc:1034] running on GCE node
W20260929 16:31:16.570127 29167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:16.570098 29169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:16.570848 29161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:16.571874 29161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:16.573072 29161 hybrid_clock.cc:648] HybridClock initialized: now 1790699476573048 us; error 45 us; skew 500 ppm
I20260929 16:31:16.574843 29161 webserver.cc:460] Webserver started at http://127.23.44.62:33981/ using document root <none> and password file <none>
I20260929 16:31:16.575384 29161 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:16.575471 29161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:16.575672 29161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:16.577730 29161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "3afb9d71bc174953bd639b4280d42f6f"
format_stamp: "Formatted at 2026-09-29 16:31:16 on dist-test-slave-mlmz"
server_key: "76d540cedf1c3363490ff319514d08e6"
server_key_iv: "bde2dff746f05b91ed4082237a90a673"
server_key_version: "encryptionkey@0"
I20260929 16:31:16.578286 29161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "3afb9d71bc174953bd639b4280d42f6f"
format_stamp: "Formatted at 2026-09-29 16:31:16 on dist-test-slave-mlmz"
server_key: "76d540cedf1c3363490ff319514d08e6"
server_key_iv: "bde2dff746f05b91ed4082237a90a673"
server_key_version: "encryptionkey@0"
I20260929 16:31:16.581712 29161 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:31:16.583961 29175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:16.585050 29161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:16.585183 29161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "3afb9d71bc174953bd639b4280d42f6f"
format_stamp: "Formatted at 2026-09-29 16:31:16 on dist-test-slave-mlmz"
server_key: "76d540cedf1c3363490ff319514d08e6"
server_key_iv: "bde2dff746f05b91ed4082237a90a673"
server_key_version: "encryptionkey@0"
I20260929 16:31:16.585335 29161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:16.624037 29161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:16.624773 29161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:16.624980 29161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:16.631649 29161 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:35719
I20260929 16:31:16.631664 29227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:35719 every 8 connection(s)
I20260929 16:31:16.632570 29161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:16.635401 29228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:16.637998 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 29161
I20260929 16:31:16.638139 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:16.638406 23728 external_mini_cluster.cc:1428] Setting key 5cff6ae4f53619496325d9337b6722cc
I20260929 16:31:16.641145 29228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f: Bootstrap starting.
I20260929 16:31:16.643249 29228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:16.644143 29228 log.cc:824] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:16.646236 29228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f: No bootstrap required, opened a new log
I20260929 16:31:16.647701 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:16.639759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.23.44.62:35719)
0929 16:31:16.640388 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:16.640403 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:16.640439 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:31:16.641034 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:16.641050 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:16.641521 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:16.641864 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:16.642299 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.643242 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.644483 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.644959 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.645120 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:16.645950 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:16.645957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:16.645969 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:16.646409 (+ 440us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:16.646463 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:16.646484 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:16.646562 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:16.646723 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 16:31:16.647035 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":135,"threads_started":1}
I20260929 16:31:16.650106 29228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afb9d71bc174953bd639b4280d42f6f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35719 } }
I20260929 16:31:16.650362 29228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:16.650417 29228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afb9d71bc174953bd639b4280d42f6f, State: Initialized, Role: FOLLOWER
I20260929 16:31:16.650763 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:31:16.650888 29228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afb9d71bc174953bd639b4280d42f6f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35719 } }
I20260929 16:31:16.651031 29228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:16.651086 29228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:16.651172 29228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:16.652222 29228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afb9d71bc174953bd639b4280d42f6f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35719 } }
I20260929 16:31:16.652557 29228 leader_election.cc:302] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3afb9d71bc174953bd639b4280d42f6f; no voters:
I20260929 16:31:16.652839 29228 leader_election.cc:288] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:16.653010 29233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:16.653304 29233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [term 1 LEADER]: Becoming Leader. State: Replica: 3afb9d71bc174953bd639b4280d42f6f, State: Running, Role: LEADER
I20260929 16:31:16.653584 29233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afb9d71bc174953bd639b4280d42f6f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35719 } }
I20260929 16:31:16.654045 29228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:16.655145 29235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3afb9d71bc174953bd639b4280d42f6f. Latest consensus state: current_term: 1 leader_uuid: "3afb9d71bc174953bd639b4280d42f6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afb9d71bc174953bd639b4280d42f6f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35719 } } }
I20260929 16:31:16.655272 29235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:16.655646 29242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:16.655624 29234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3afb9d71bc174953bd639b4280d42f6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afb9d71bc174953bd639b4280d42f6f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35719 } } }
I20260929 16:31:16.655797 29234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:16.658892 29242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:16.664566 29242 catalog_manager.cc:1347] Generated new cluster ID: 6d58499c319748bbbd2551e98342e0f6
I20260929 16:31:16.664641 29242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:16.703125 29242 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:16.704033 29242 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:16.707727 29242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3afb9d71bc174953bd639b4280d42f6f: Generated new TSK 0
I20260929 16:31:16.708350 29242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:31:16.743077 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:16.715530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.23.44.62:35719)
0929 16:31:16.715646 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:16.715652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:16.715826 (+ 174us) server_negotiation.cc:407] Connection header received
0929 16:31:16.716030 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:16.716033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:16.716077 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:16.716187 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:31:16.716423 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.717443 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.718310 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.718536 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.718647 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:16.718954 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:31:16.742609 (+ 23655us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 16:31:16.742898 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57112: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 16:31:16.743163 29257 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:16.715475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:35719 (local address 127.0.0.1:57112)
0929 16:31:16.715742 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:16.715754 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:16.715874 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:16.716188 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:16.716192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:16.716200 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:31:16.716310 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:16.716319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.717566 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.717569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:16.718188 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:16.718196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:16.718659 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:16.718663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:16.718754 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:16.718782 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:31:16.742816 (+ 24034us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 16:31:16.742994 (+ 178us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.44.62:35719: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
W20260929 16:31:16.743161 29231 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57112: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260929 16:31:16.743938 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 29161
2026-09-29T16:31:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (328 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (116 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:17 dist-test-slave-mlmz krb5kdc[29269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:17 dist-test-slave-mlmz krb5kdc[29269](info): set up 2 sockets
Sep 29 16:31:17 dist-test-slave-mlmz krb5kdc[29269](info): commencing operation
krb5kdc: starting...
W20260929 16:31:18.041534 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:31:18 dist-test-slave-mlmz krb5kdc[29269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:31:18.160434 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:35501
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:35501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:18.265002 29282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:18.265249 29282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:18.265379 29282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:18.269434 29282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:18.269508 29282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:18.269546 29282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:18.269603 29282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:18.269642 29282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:18.274652 29282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:35501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:35501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:18.275601 29282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:18.276546 29282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:18.284498 29287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:18.284498 29288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:18.284466 29290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:18.284776 29282 server_base.cc:1034] running on GCE node
I20260929 16:31:18.285427 29282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:18.286448 29282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:18.287639 29282 hybrid_clock.cc:648] HybridClock initialized: now 1790699478287627 us; error 24 us; skew 500 ppm
Sep 29 16:31:18 dist-test-slave-mlmz krb5kdc[29269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699478, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:18.290354 29282 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:31:18.291173 29282 webserver.cc:460] Webserver started at http://127.23.44.62:34667/ using document root <none> and password file <none>
I20260929 16:31:18.291692 29282 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:18.291779 29282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:18.292025 29282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:18.293988 29282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "487de1e1e06c48d88f22055d477be675"
format_stamp: "Formatted at 2026-09-29 16:31:18 on dist-test-slave-mlmz"
server_key: "f7573e0c51acfd587024c7e892699f8b"
server_key_iv: "570f201034ab4ff25a268cd0787d904a"
server_key_version: "encryptionkey@0"
I20260929 16:31:18.294530 29282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "487de1e1e06c48d88f22055d477be675"
format_stamp: "Formatted at 2026-09-29 16:31:18 on dist-test-slave-mlmz"
server_key: "f7573e0c51acfd587024c7e892699f8b"
server_key_iv: "570f201034ab4ff25a268cd0787d904a"
server_key_version: "encryptionkey@0"
I20260929 16:31:18.297931 29282 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:31:18.300226 29297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:18.301409 29282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:18.301538 29282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "487de1e1e06c48d88f22055d477be675"
format_stamp: "Formatted at 2026-09-29 16:31:18 on dist-test-slave-mlmz"
server_key: "f7573e0c51acfd587024c7e892699f8b"
server_key_iv: "570f201034ab4ff25a268cd0787d904a"
server_key_version: "encryptionkey@0"
I20260929 16:31:18.301640 29282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:18.322011 29282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:18.325160 29282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:18.325378 29282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:18.332206 29282 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:35501
I20260929 16:31:18.332214 29349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:35501 every 8 connection(s)
I20260929 16:31:18.333110 29282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:18.335888 29350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:18.335943 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 29282
I20260929 16:31:18.336056 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:18.336371 23728 external_mini_cluster.cc:1428] Setting key dd7d14267b86d7725a0eedc2b843b5a1
I20260929 16:31:18.342403 29350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675: Bootstrap starting.
I20260929 16:31:18.345046 29350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675: Neither blocks nor log segments found. Creating new log.
Sep 29 16:31:18 dist-test-slave-mlmz krb5kdc[29269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:18.345888 29350 log.cc:824] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:18.347595 29350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675: No bootstrap required, opened a new log
I20260929 16:31:18.350796 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:18.337719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.23.44.62:35501)
0929 16:31:18.338150 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:18.338167 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:18.338202 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:31:18.338917 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:18.338933 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:18.339269 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:18.339548 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:18.340438 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:18.341562 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:18.342676 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:18.343171 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:18.343303 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:18.346248 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:18.346253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:18.346266 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:18.346291 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:18.348094 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:18.348527 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:18.348537 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:18.348541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:18.348649 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:18.348930 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:18.348936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:18.348941 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:18.349370 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:18.349570 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:18.349852 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 16:31:18.350125 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":144,"threads_started":1}
I20260929 16:31:18.351171 29350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487de1e1e06c48d88f22055d477be675" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35501 } }
I20260929 16:31:18.351399 29350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:18.351475 29350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487de1e1e06c48d88f22055d477be675, State: Initialized, Role: FOLLOWER
I20260929 16:31:18.351920 29350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487de1e1e06c48d88f22055d477be675" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35501 } }
I20260929 16:31:18.352068 29350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:18.352129 29350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:18.352218 29350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:18.353332 29350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487de1e1e06c48d88f22055d477be675" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35501 } }
I20260929 16:31:18.353612 29350 leader_election.cc:302] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 487de1e1e06c48d88f22055d477be675; no voters:
I20260929 16:31:18.353844 29350 leader_election.cc:288] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:18.353986 29355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:18.354206 29355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [term 1 LEADER]: Becoming Leader. State: Replica: 487de1e1e06c48d88f22055d477be675, State: Running, Role: LEADER
I20260929 16:31:18.354543 29355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487de1e1e06c48d88f22055d477be675" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35501 } }
I20260929 16:31:18.355067 29350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:18.356143 29356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "487de1e1e06c48d88f22055d477be675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487de1e1e06c48d88f22055d477be675" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35501 } } }
I20260929 16:31:18.356283 29356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:18.356602 29357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 487de1e1e06c48d88f22055d477be675. Latest consensus state: current_term: 1 leader_uuid: "487de1e1e06c48d88f22055d477be675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487de1e1e06c48d88f22055d477be675" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 35501 } } }
I20260929 16:31:18.356671 29364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:18.356752 29357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:18.359450 29364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:18.364755 29364 catalog_manager.cc:1347] Generated new cluster ID: 872a06ca34684a28b237f4ed7e0f98db
I20260929 16:31:18.364825 29364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:18.387799 29364 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:18.388882 29364 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:18.398221 29364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 487de1e1e06c48d88f22055d477be675: Generated new TSK 0
I20260929 16:31:18.398943 29364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:31:18.426419 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:35501
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:18.541787 29376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:18.542115 29376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:18.542194 29376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:18.546546 29376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:18.546654 29376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:18.546818 29376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:18.555389 29376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:35501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:18.556811 29376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:18.557875 29376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:18.567039 29384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:18.567054 29381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:18.567113 29376 server_base.cc:1034] running on GCE node
W20260929 16:31:18.567054 29382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:18.567961 29376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:18.568667 29376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:18.569870 29376 hybrid_clock.cc:648] HybridClock initialized: now 1790699478569849 us; error 34 us; skew 500 ppm
Sep 29 16:31:18 dist-test-slave-mlmz krb5kdc[29269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699478, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:18.581352 29376 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:31:18.582271 29376 webserver.cc:460] Webserver started at http://127.23.44.1:33185/ using document root <none> and password file <none>
I20260929 16:31:18.582871 29376 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:18.582968 29376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:18.583195 29376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:18.585201 29376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "6b779b982a494ee996fffc56bc24c0f6"
format_stamp: "Formatted at 2026-09-29 16:31:18 on dist-test-slave-mlmz"
server_key: "89ec01eaa149f5d9c96f0ccee38c6923"
server_key_iv: "b1e212e19a18f12760119b59465c1920"
server_key_version: "encryptionkey@0"
I20260929 16:31:18.585825 29376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "6b779b982a494ee996fffc56bc24c0f6"
format_stamp: "Formatted at 2026-09-29 16:31:18 on dist-test-slave-mlmz"
server_key: "89ec01eaa149f5d9c96f0ccee38c6923"
server_key_iv: "b1e212e19a18f12760119b59465c1920"
server_key_version: "encryptionkey@0"
I20260929 16:31:18.589406 29376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:31:18.591792 29391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:18.593080 29376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:18.593235 29376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "6b779b982a494ee996fffc56bc24c0f6"
format_stamp: "Formatted at 2026-09-29 16:31:18 on dist-test-slave-mlmz"
server_key: "89ec01eaa149f5d9c96f0ccee38c6923"
server_key_iv: "b1e212e19a18f12760119b59465c1920"
server_key_version: "encryptionkey@0"
I20260929 16:31:18.593416 29376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:18.635905 29376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:18.638976 29376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:18.639196 29376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:18.639796 29376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:18.640673 29376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:18.640748 29376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:18.640825 29376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:18.640865 29376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:18.650151 29376 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:32917
I20260929 16:31:18.650178 29504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:32917 every 8 connection(s)
I20260929 16:31:18.651050 29376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:18.653609 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 29376
I20260929 16:31:18.653728 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:18.654029 23728 external_mini_cluster.cc:1428] Setting key a3c62bc08b63dff3e34526e4c9a64309
Sep 29 16:31:18 dist-test-slave-mlmz krb5kdc[29269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699478, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
I20260929 16:31:18.667037 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:18.652852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:51795 (local address 127.23.44.62:35501)
0929 16:31:18.652975 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:18.652979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:18.654053 (+ 1074us) server_negotiation.cc:407] Connection header received
0929 16:31:18.655823 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:18.655826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:18.655878 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:18.655959 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:18.657744 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:18.658337 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:18.659691 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:18.660053 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:18.660163 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:18.662766 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:18.662770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:18.662773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:18.662806 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:18.664422 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:18.664967 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:18.664972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:18.664974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:18.665047 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:18.665541 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:18.665544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:18.665546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:18.665727 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:18.665853 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:18.666569 (+ 716us) server_negotiation.cc:299] Negotiation successful
0929 16:31:18.666759 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 16:31:18.667549 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:18.653241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:35501 (local address 127.23.44.1:51795)
0929 16:31:18.653895 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:18.653950 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:18.655557 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:18.656177 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:18.656189 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:18.656768 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:18.657583 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:18.657599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:18.658476 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:18.658481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:18.659562 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:18.659572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:18.660199 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:18.660203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:18.660375 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:18.661037 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:18.661054 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:18.662598 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:18.664548 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:18.664554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:18.664569 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:18.664832 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:18.665187 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:18.665190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:18.665192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:18.665421 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:18.665936 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:18.665943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:18.666263 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 16:31:18.666451 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 16:31:18.666716 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":158,"threads_started":1}
I20260929 16:31:18.668792 29505 heartbeater.cc:347] Connected to a master server at 127.23.44.62:35501
I20260929 16:31:18.669037 29505 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:18.669575 29505 heartbeater.cc:511] Master 127.23.44.62:35501 requested a full tablet report, sending...
I20260929 16:31:18.671037 29314 ts_manager.cc:194] Registered new tserver with Master: 6b779b982a494ee996fffc56bc24c0f6 (127.23.44.1:32917)
I20260929 16:31:18.672261 29314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:51795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:31:18.684424 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:35501
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:18.792560 29510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:18.792788 29510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:18.792829 29510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:18.796856 29510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:18.796907 29510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:18.796964 29510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:18.802359 29510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:35501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:18.803263 29510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:18.804193 29510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:18.813337 29516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:18.813408 29510 server_base.cc:1034] running on GCE node
W20260929 16:31:18.813419 29518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:18.813337 29515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:18.814014 29510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:18.814572 29510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:18.815722 29510 hybrid_clock.cc:648] HybridClock initialized: now 1790699478815706 us; error 30 us; skew 500 ppm
Sep 29 16:31:18 dist-test-slave-mlmz krb5kdc[29269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699478, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:18.818465 29510 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:31:18.819294 29510 webserver.cc:460] Webserver started at http://127.23.44.2:33273/ using document root <none> and password file <none>
I20260929 16:31:18.819816 29510 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:18.819872 29510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:18.820024 29510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:18.821884 29510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "40e1b1d10d774f4a94054260708a2f33"
format_stamp: "Formatted at 2026-09-29 16:31:18 on dist-test-slave-mlmz"
server_key: "dd9835b706512fcc4629119c0b7750ff"
server_key_iv: "95fdb359745b4e81e8d54989f1714832"
server_key_version: "encryptionkey@0"
I20260929 16:31:18.822365 29510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "40e1b1d10d774f4a94054260708a2f33"
format_stamp: "Formatted at 2026-09-29 16:31:18 on dist-test-slave-mlmz"
server_key: "dd9835b706512fcc4629119c0b7750ff"
server_key_iv: "95fdb359745b4e81e8d54989f1714832"
server_key_version: "encryptionkey@0"
I20260929 16:31:18.825724 29510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:31:18.828004 29525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:18.829190 29510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:18.829346 29510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "40e1b1d10d774f4a94054260708a2f33"
format_stamp: "Formatted at 2026-09-29 16:31:18 on dist-test-slave-mlmz"
server_key: "dd9835b706512fcc4629119c0b7750ff"
server_key_iv: "95fdb359745b4e81e8d54989f1714832"
server_key_version: "encryptionkey@0"
I20260929 16:31:18.829453 29510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:18.853070 29510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:18.856189 29510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:18.856389 29510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:18.856946 29510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:18.857722 29510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:18.857767 29510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:18.857805 29510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:18.857820 29510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:18.866417 29510 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:35543
I20260929 16:31:18.866448 29638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:35543 every 8 connection(s)
I20260929 16:31:18.867233 29510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:18.870250 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 29510
I20260929 16:31:18.870335 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:18.870626 23728 external_mini_cluster.cc:1428] Setting key f7b21f9d2c7b05e66c033bb6215d7ad5
Sep 29 16:31:18 dist-test-slave-mlmz krb5kdc[29269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699478, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:31:18.880761 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:18.868838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:53071 (local address 127.23.44.62:35501)
0929 16:31:18.868970 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:18.868973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:18.869806 (+ 833us) server_negotiation.cc:407] Connection header received
0929 16:31:18.871336 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:18.871339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:18.871392 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:18.871473 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:18.872650 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:18.873246 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:18.874359 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:18.874776 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:18.874899 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:18.877172 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:18.877177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:18.877180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:18.877210 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:18.878850 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:18.879323 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:18.879326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:18.879328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:18.879416 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:18.879733 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:18.879738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:18.879742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:18.879991 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:18.880092 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:18.880510 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 16:31:18.880603 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:31:18.881727 29641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:18.869206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:35501 (local address 127.23.44.2:53071)
0929 16:31:18.869672 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:18.869710 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:18.871137 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:18.871631 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:18.871637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:18.872062 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:18.872515 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:18.872527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:18.873405 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:18.873407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:18.874236 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:18.874245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:18.874916 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:18.874919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:18.875047 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:18.875640 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:18.875655 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:18.877012 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:18.878969 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:18.878974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:18.878984 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:18.879217 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:18.879509 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:18.879511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:18.879513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:18.879634 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:18.880091 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:18.880095 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:18.880302 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 16:31:18.880457 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 16:31:18.880768 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
I20260929 16:31:18.882882 29639 heartbeater.cc:347] Connected to a master server at 127.23.44.62:35501
I20260929 16:31:18.883098 29639 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:18.883572 29639 heartbeater.cc:511] Master 127.23.44.62:35501 requested a full tablet report, sending...
I20260929 16:31:18.884586 29314 ts_manager.cc:194] Registered new tserver with Master: 40e1b1d10d774f4a94054260708a2f33 (127.23.44.2:35543)
I20260929 16:31:18.885454 29314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:53071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:31:18.897061 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:35501
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:19.003947 29644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:19.004184 29644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:19.004225 29644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:19.008396 29644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:19.008451 29644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:19.008545 29644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:19.013638 29644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:35501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:19.014583 29644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:19.015535 29644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:19.025209 29649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:19.025179 29650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:19.025189 29652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:19.025389 29644 server_base.cc:1034] running on GCE node
I20260929 16:31:19.025825 29644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:19.026388 29644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:19.027542 29644 hybrid_clock.cc:648] HybridClock initialized: now 1790699479027528 us; error 24 us; skew 500 ppm
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699479, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:19.030205 29644 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:31:19.030928 29644 webserver.cc:460] Webserver started at http://127.23.44.3:45777/ using document root <none> and password file <none>
I20260929 16:31:19.031399 29644 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:19.031453 29644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:19.031600 29644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:19.033643 29644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "667a6a4f0c084590b94c01f13e5e9d51"
format_stamp: "Formatted at 2026-09-29 16:31:19 on dist-test-slave-mlmz"
server_key: "4cde12f5b919c0b781ed98fe5e8f4119"
server_key_iv: "e3f18d706f010f64769122a67f0fe825"
server_key_version: "encryptionkey@0"
I20260929 16:31:19.034121 29644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "667a6a4f0c084590b94c01f13e5e9d51"
format_stamp: "Formatted at 2026-09-29 16:31:19 on dist-test-slave-mlmz"
server_key: "4cde12f5b919c0b781ed98fe5e8f4119"
server_key_iv: "e3f18d706f010f64769122a67f0fe825"
server_key_version: "encryptionkey@0"
I20260929 16:31:19.037519 29644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:31:19.039716 29659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:19.040949 29644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:19.041090 29644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "667a6a4f0c084590b94c01f13e5e9d51"
format_stamp: "Formatted at 2026-09-29 16:31:19 on dist-test-slave-mlmz"
server_key: "4cde12f5b919c0b781ed98fe5e8f4119"
server_key_iv: "e3f18d706f010f64769122a67f0fe825"
server_key_version: "encryptionkey@0"
I20260929 16:31:19.041199 29644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:19.056679 29644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:19.059517 29644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:19.059705 29644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:19.060236 29644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:19.060976 29644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:19.061043 29644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:19.061113 29644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:19.061156 29644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:19.069285 29644 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:33763
I20260929 16:31:19.069346 29772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:33763 every 8 connection(s)
I20260929 16:31:19.070124 29644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:19.072747 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 29644
I20260929 16:31:19.072870 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:19.073179 23728 external_mini_cluster.cc:1428] Setting key 66f438df9333ea9dabc7b2d474a56b33
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699479, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:19.083536 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:19.071856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:41495 (local address 127.23.44.62:35501)
0929 16:31:19.072002 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:19.072007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:19.072938 (+ 931us) server_negotiation.cc:407] Connection header received
0929 16:31:19.074738 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:19.074741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:19.074795 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:19.074900 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:19.076039 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:19.076566 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:19.077813 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:19.078028 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:19.078116 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:19.080344 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:19.080347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:19.080349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:19.080372 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:19.081850 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:19.082309 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:19.082311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:19.082313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:19.082383 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:19.082665 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:19.082668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:19.082669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:19.082842 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:19.082918 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:19.083326 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 16:31:19.083412 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:31:19.084259 29775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:19.072275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:35501 (local address 127.23.44.3:41495)
0929 16:31:19.072772 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:19.072828 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:19.074529 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:19.075073 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:19.075080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:19.075469 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:19.075905 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:19.075917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:19.076685 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:19.076689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:19.077682 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:19.077693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:19.078154 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:19.078156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:19.078284 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:19.078790 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:19.078805 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:19.080200 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:19.081971 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:19.081975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:19.081985 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:19.082218 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:19.082460 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:19.082462 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:19.082463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:19.082579 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:19.082934 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:19.082938 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:19.083135 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 16:31:19.083280 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 16:31:19.083497 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":106,"threads_started":1}
I20260929 16:31:19.085400 29773 heartbeater.cc:347] Connected to a master server at 127.23.44.62:35501
I20260929 16:31:19.085614 29773 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:19.086160 29773 heartbeater.cc:511] Master 127.23.44.62:35501 requested a full tablet report, sending...
I20260929 16:31:19.087113 29314 ts_manager.cc:194] Registered new tserver with Master: 667a6a4f0c084590b94c01f13e5e9d51 (127.23.44.3:33763)
I20260929 16:31:19.087666 29314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:41495
I20260929 16:31:19.098389 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/test-admin.keytab.
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:19.113421 23728 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 16:31:19.115957 29780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:19.114671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60512 (local address 127.23.44.1:32917)
0929 16:31:19.114922 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:19.114942 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:19.114989 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:31:19.115035 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:19.115039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:19.115207 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:19.115321 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:19.115780 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60512: BlockingRecv error: recv got EOF from 127.0.0.1:60512 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":73,"threads_started":1}
W20260929 16:31:19.115999 23728 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:19.117069 23728 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:19.118114 23728 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:31:19.127751 29780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:19.118867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.23.44.1:32917)
0929 16:31:19.118993 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:19.118997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:19.119097 (+ 100us) server_negotiation.cc:407] Connection header received
0929 16:31:19.119240 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:19.119243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:19.119292 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:19.119377 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:19.119891 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:19.120640 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:19.121531 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:19.121893 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:19.122013 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:19.124192 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:19.124206 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:19.124217 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:19.124238 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:19.125889 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:19.126439 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:19.126456 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:19.126457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:19.126534 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:19.126832 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:19.126837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:19.126838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:19.127063 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:19.127254 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:19.127484 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:31:19.127638 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 16:31:19.131134 29780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:19.130200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.23.44.1:32917)
0929 16:31:19.130316 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:19.130320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:19.130418 (+ 98us) server_negotiation.cc:407] Connection header received
0929 16:31:19.130554 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:19.130557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:19.130597 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:19.130678 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:19.131045 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60532: BlockingRecv error: recv got EOF from 127.0.0.1:60532 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 16:31:19.131283 23728 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:19.132289 23728 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:19.133189 23728 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:31:19.141379 29780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:19.134036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.23.44.1:32917)
0929 16:31:19.134151 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:19.134155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:19.134266 (+ 111us) server_negotiation.cc:407] Connection header received
0929 16:31:19.134399 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:19.134402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:19.134438 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:19.134525 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:19.134992 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:19.135503 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:19.136262 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:19.136507 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:19.136611 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:19.138620 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:19.138623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:19.138626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:19.138654 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:19.139828 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:19.140244 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:19.140248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:19.140249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:19.140325 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:19.140597 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:19.140600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:19.140602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:19.140751 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:19.140873 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:19.141107 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 16:31:19.141215 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 16:31:19.143662 29780 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:19.142831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.23.44.1:32917)
0929 16:31:19.142931 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:19.142934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:19.143033 (+ 99us) server_negotiation.cc:407] Connection header received
0929 16:31:19.143124 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:19.143126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:19.143154 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:19.143213 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:19.143569 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60550: BlockingRecv error: recv got EOF from 127.0.0.1:60550 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 16:31:19.143795 23728 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699450427356-23728-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:19.144760 23728 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:19.145730 23728 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:31:19.153424 29780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:19.146388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.23.44.1:32917)
0929 16:31:19.146492 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:19.146496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:19.146620 (+ 124us) server_negotiation.cc:407] Connection header received
0929 16:31:19.146687 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:19.146688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:19.146718 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:19.146786 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:19.147183 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:19.147626 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:19.148363 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:19.148554 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:19.148623 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:19.150768 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:19.150771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:19.150773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:19.150792 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:19.151980 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:19.152386 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:19.152389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:19.152390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:19.152458 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:19.152692 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:19.152694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:19.152695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:19.152835 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:19.152938 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:19.153120 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 16:31:19.153242 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:31:19.154320 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 29376
I20260929 16:31:19.159950 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 29510
I20260929 16:31:19.165180 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 29644
I20260929 16:31:19.170234 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 29282
2026-09-29T16:31:19Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2192 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29793](info): set up 2 sockets
Sep 29 16:31:19 dist-test-slave-mlmz krb5kdc[29793](info): commencing operation
krb5kdc: starting...
W20260929 16:31:20.233613 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:31:20 dist-test-slave-mlmz krb5kdc[29793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.62.keytab.
Entry for principal oryx/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.62.keytab.
I20260929 16:31:20.358364 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:44913
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.62.keytab
--principal=oryx/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:44913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:20.465757 29806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:20.466032 29806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:20.466099 29806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:20.470429 29806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:20.470508 29806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:20.470546 29806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:20.470602 29806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:20.470638 29806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:20.475718 29806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:44913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.62.keytab
--principal=oryx/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:44913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:20.476709 29806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:20.477713 29806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:31:20.485904 29806 server_base.cc:1034] running on GCE node
W20260929 16:31:20.485859 29811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:20.485859 29812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:20.485867 29814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:20.486728 29806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:20.487751 29806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:20.488921 29806 hybrid_clock.cc:648] HybridClock initialized: now 1790699480488905 us; error 29 us; skew 500 ppm
Sep 29 16:31:20 dist-test-slave-mlmz krb5kdc[29793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699480, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:20.491886 29806 init.cc:377] Logged in from keytab as oryx/127.23.44.62@KRBTEST.COM (short username oryx)
I20260929 16:31:20.492833 29806 webserver.cc:460] Webserver started at http://127.23.44.62:43895/ using document root <none> and password file <none>
I20260929 16:31:20.493425 29806 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:20.493499 29806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:20.493705 29806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:20.495683 29806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "939af390c9f94d3c93c53b23f0754cc2"
format_stamp: "Formatted at 2026-09-29 16:31:20 on dist-test-slave-mlmz"
server_key: "561fd06a69dbdf4e57b27b8587eb8eb8"
server_key_iv: "52a9df3183b2fc72ef250f10dc73473f"
server_key_version: "encryptionkey@0"
I20260929 16:31:20.496236 29806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "939af390c9f94d3c93c53b23f0754cc2"
format_stamp: "Formatted at 2026-09-29 16:31:20 on dist-test-slave-mlmz"
server_key: "561fd06a69dbdf4e57b27b8587eb8eb8"
server_key_iv: "52a9df3183b2fc72ef250f10dc73473f"
server_key_version: "encryptionkey@0"
I20260929 16:31:20.499861 29806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 16:31:20.502241 29821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:20.503541 29806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:20.503682 29806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "939af390c9f94d3c93c53b23f0754cc2"
format_stamp: "Formatted at 2026-09-29 16:31:20 on dist-test-slave-mlmz"
server_key: "561fd06a69dbdf4e57b27b8587eb8eb8"
server_key_iv: "52a9df3183b2fc72ef250f10dc73473f"
server_key_version: "encryptionkey@0"
I20260929 16:31:20.503791 29806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:20.539979 29806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:20.543280 29806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:20.543497 29806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:20.552682 29806 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:44913
I20260929 16:31:20.552986 29873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:44913 every 8 connection(s)
I20260929 16:31:20.553869 29806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:20.554363 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 29806
I20260929 16:31:20.554494 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:20.554844 23728 external_mini_cluster.cc:1428] Setting key 7c35fa4043f1f5647d9851afadc1a492
I20260929 16:31:20.557386 29874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:20.562978 29874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2: Bootstrap starting.
Sep 29 16:31:20 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
I20260929 16:31:20.565445 29874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:20.566300 29874 log.cc:824] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:20.568761 29874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2: No bootstrap required, opened a new log
I20260929 16:31:20.574007 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:20.556198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.23.44.62:44913)
0929 16:31:20.556750 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:20.556773 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:20.556817 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:31:20.557858 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:20.557881 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:20.558362 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:20.558709 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:20.559488 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:20.560475 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:20.561364 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:20.561829 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:20.562018 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:20.564711 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:20.564716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:20.564728 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:20.564754 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:20.570536 (+ 5782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:20.571253 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:20.571258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:20.571264 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:20.571402 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:20.571705 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:20.571710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:20.571712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:20.572131 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:20.572375 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:20.572673 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 16:31:20.573007 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":132,"threads_started":1}
I20260929 16:31:20.574086 29874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } }
I20260929 16:31:20.574342 29874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:20.574396 29874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939af390c9f94d3c93c53b23f0754cc2, State: Initialized, Role: FOLLOWER
I20260929 16:31:20.574836 29874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } }
I20260929 16:31:20.574992 29874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:20.575049 29874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:20.576145 29874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:20.577077 29874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } }
I20260929 16:31:20.577482 29874 leader_election.cc:302] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 939af390c9f94d3c93c53b23f0754cc2; no voters:
I20260929 16:31:20.577795 29874 leader_election.cc:288] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:20.578019 29879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:20.578289 29879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 1 LEADER]: Becoming Leader. State: Replica: 939af390c9f94d3c93c53b23f0754cc2, State: Running, Role: LEADER
I20260929 16:31:20.578614 29879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } }
I20260929 16:31:20.579131 29874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:20.580255 29880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } }
I20260929 16:31:20.580376 29880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:20.580760 29888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:20.580756 29881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 939af390c9f94d3c93c53b23f0754cc2. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } }
I20260929 16:31:20.580914 29881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:20.583640 29888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:20.589443 29888 catalog_manager.cc:1347] Generated new cluster ID: e33f563b1bbe46d5a0c2b73bb61906b7
I20260929 16:31:20.589560 29888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:20.604854 29888 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:20.605836 29888 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:20.614617 29888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2: Generated new TSK 0
I20260929 16:31:20.615275 29888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab.
Entry for principal oryx/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab.
I20260929 16:31:20.653191 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:44913
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab
--principal=oryx/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:20.780850 29900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:20.781091 29900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:20.781133 29900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:20.785040 29900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:20.785095 29900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:20.785184 29900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:20.790381 29900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab
--principal=oryx/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:20.791370 29900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:20.792372 29900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:31:20.801855 29900 server_base.cc:1034] running on GCE node
W20260929 16:31:20.801872 29908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:20.801882 29906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:20.801862 29905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:20.802559 29900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:20.803237 29900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:20.804467 29900 hybrid_clock.cc:648] HybridClock initialized: now 1790699480804454 us; error 29 us; skew 500 ppm
Sep 29 16:31:20 dist-test-slave-mlmz krb5kdc[29793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699480, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:20.807523 29900 init.cc:377] Logged in from keytab as oryx/127.23.44.1@KRBTEST.COM (short username oryx)
I20260929 16:31:20.808530 29900 webserver.cc:460] Webserver started at http://127.23.44.1:41531/ using document root <none> and password file <none>
I20260929 16:31:20.809118 29900 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:20.809216 29900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:20.809481 29900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:20.811537 29900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "30dfcbab3ef543848c69b4734509af52"
format_stamp: "Formatted at 2026-09-29 16:31:20 on dist-test-slave-mlmz"
server_key: "e10f3625a5222b8efefa1d683e5c92c4"
server_key_iv: "9e4d215ae8a393f0964a3c2749952538"
server_key_version: "encryptionkey@0"
I20260929 16:31:20.812160 29900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "30dfcbab3ef543848c69b4734509af52"
format_stamp: "Formatted at 2026-09-29 16:31:20 on dist-test-slave-mlmz"
server_key: "e10f3625a5222b8efefa1d683e5c92c4"
server_key_iv: "9e4d215ae8a393f0964a3c2749952538"
server_key_version: "encryptionkey@0"
I20260929 16:31:20.816521 29900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:31:20.819061 29915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:20.820245 29900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:31:20.820389 29900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "30dfcbab3ef543848c69b4734509af52"
format_stamp: "Formatted at 2026-09-29 16:31:20 on dist-test-slave-mlmz"
server_key: "e10f3625a5222b8efefa1d683e5c92c4"
server_key_iv: "9e4d215ae8a393f0964a3c2749952538"
server_key_version: "encryptionkey@0"
I20260929 16:31:20.820509 29900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:20.832935 29900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:20.836145 29900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:20.836339 29900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:20.836886 29900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:20.837733 29900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:20.837802 29900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:20.837870 29900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:20.837922 29900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:20.846473 29900 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:36317
I20260929 16:31:20.846491 30028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:36317 every 8 connection(s)
I20260929 16:31:20.847321 29900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:20.849998 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 29900
I20260929 16:31:20.850143 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:20.850498 23728 external_mini_cluster.cc:1428] Setting key cb251c0f8f0801a4d4d037421476b8ee
Sep 29 16:31:20 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699480, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.23.44.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.2@KRBTEST.COM" created.
I20260929 16:31:20.862916 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:20.849062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:46455 (local address 127.23.44.62:44913)
0929 16:31:20.849226 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:20.849232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:20.850208 (+ 976us) server_negotiation.cc:407] Connection header received
0929 16:31:20.852008 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:20.852018 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:20.852059 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:20.852126 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:20.853890 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:20.854471 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:20.855668 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:20.856096 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:20.856210 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:20.859005 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:20.859011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:20.859017 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:20.859052 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:20.860711 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:20.861280 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:20.861287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:20.861292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:20.861399 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:20.861745 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:20.861748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:20.861750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:20.861945 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:20.862057 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:20.862608 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 16:31:20.862745 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:31:20.863853 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:20.849472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.1:46455)
0929 16:31:20.850038 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:20.850105 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:20.851752 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:20.852313 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:20.852325 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:20.852901 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:20.853664 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:20.853690 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:20.854604 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:20.854607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:20.855546 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:20.855554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:20.856267 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:20.856271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:20.856421 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:20.857066 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:20.857083 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:20.858828 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:20.860831 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:20.860842 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:20.860854 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:20.861102 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:20.861506 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:20.861514 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:20.861515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:20.861641 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:20.862055 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:20.862062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:20.862355 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 16:31:20.862539 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 16:31:20.862900 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":99,"threads_started":1}
I20260929 16:31:20.865314 30029 heartbeater.cc:347] Connected to a master server at 127.23.44.62:44913
I20260929 16:31:20.865583 30029 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:20.866102 30029 heartbeater.cc:511] Master 127.23.44.62:44913 requested a full tablet report, sending...
I20260929 16:31:20.867795 29838 ts_manager.cc:194] Registered new tserver with Master: 30dfcbab3ef543848c69b4734509af52 (127.23.44.1:36317)
I20260929 16:31:20.869585 29838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:46455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab.
Entry for principal oryx/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab.
I20260929 16:31:20.881469 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:44913
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab
--principal=oryx/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:20.995157 30034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:20.995409 30034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:20.995452 30034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:20.999536 30034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:20.999588 30034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:20.999642 30034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:21.004633 30034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab
--principal=oryx/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:21.005602 30034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:21.006515 30034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:21.016197 30040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:21.016206 30039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:21.016412 30034 server_base.cc:1034] running on GCE node
W20260929 16:31:21.016335 30042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:21.016892 30034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:21.017515 30034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:21.018683 30034 hybrid_clock.cc:648] HybridClock initialized: now 1790699481018667 us; error 27 us; skew 500 ppm
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:21.021495 30034 init.cc:377] Logged in from keytab as oryx/127.23.44.2@KRBTEST.COM (short username oryx)
I20260929 16:31:21.022383 30034 webserver.cc:460] Webserver started at http://127.23.44.2:44373/ using document root <none> and password file <none>
I20260929 16:31:21.022959 30034 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:21.023079 30034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:21.023325 30034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:21.025396 30034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "ba8e8b9ff7ccc9757baf5d133d0d27e1"
server_key_iv: "4e89a42190afa75289915a0f91bd8a31"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.025943 30034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "ba8e8b9ff7ccc9757baf5d133d0d27e1"
server_key_iv: "4e89a42190afa75289915a0f91bd8a31"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.029553 30034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:31:21.031801 30049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:21.033049 30034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:21.033177 30034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "ba8e8b9ff7ccc9757baf5d133d0d27e1"
server_key_iv: "4e89a42190afa75289915a0f91bd8a31"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.033303 30034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:21.043622 30034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:21.046670 30034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:21.046882 30034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:21.047458 30034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:21.048210 30034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:21.048280 30034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:21.048343 30034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:21.048390 30034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:21.056794 30034 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:38115
I20260929 16:31:21.056850 30162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:38115 every 8 connection(s)
I20260929 16:31:21.057711 30034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
I20260929 16:31:21.067663 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 30034
I20260929 16:31:21.067763 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:21.068082 23728 external_mini_cluster.cc:1428] Setting key 90a4a1b5dde6e35f5185773917270dcb
I20260929 16:31:21.071717 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.059559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:53321 (local address 127.23.44.62:44913)
0929 16:31:21.059704 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.059709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.060592 (+ 883us) server_negotiation.cc:407] Connection header received
0929 16:31:21.061937 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.061940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.061987 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.062063 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.063221 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.063772 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.064962 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.065188 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.065408 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.067946 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.067949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.067951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.067973 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.069660 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.070187 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.070192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.070194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.070274 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.070611 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.070615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.070617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.070844 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.070953 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.071436 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.071554 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:31:21.072489 30165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.059958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.2:53321)
0929 16:31:21.060444 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.060486 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.061708 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.062228 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.062234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.062614 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.063082 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.063094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.063897 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.063901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.064831 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.064840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.065465 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.065468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.065607 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.066263 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.066278 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.067735 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.069810 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.069816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.069830 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.070082 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.070366 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.070371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.070373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.070512 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.070952 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.070958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.071200 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.071395 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.071700 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":102,"threads_started":1}
I20260929 16:31:21.073594 30163 heartbeater.cc:347] Connected to a master server at 127.23.44.62:44913
I20260929 16:31:21.073869 30163 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:21.074445 30163 heartbeater.cc:511] Master 127.23.44.62:44913 requested a full tablet report, sending...
I20260929 16:31:21.075598 29838 ts_manager.cc:194] Registered new tserver with Master: c9b7e6cec6524c3082a32c7df0f4ab6c (127.23.44.2:38115)
I20260929 16:31:21.076205 29838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:53321
WARNING: no policy specified for oryx/127.23.44.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab.
Entry for principal oryx/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab.
I20260929 16:31:21.097318 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:44913
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab
--principal=oryx/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:21.210636 30168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:21.210893 30168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:21.210937 30168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:21.215098 30168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:21.215152 30168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:21.215209 30168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:21.220232 30168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab
--principal=oryx/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:21.221175 30168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:21.222230 30168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:21.232012 30174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:21.232007 30176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:21.232026 30173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:21.232187 30168 server_base.cc:1034] running on GCE node
I20260929 16:31:21.232825 30168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:21.233498 30168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:21.234653 30168 hybrid_clock.cc:648] HybridClock initialized: now 1790699481234636 us; error 30 us; skew 500 ppm
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:21.237509 30168 init.cc:377] Logged in from keytab as oryx/127.23.44.3@KRBTEST.COM (short username oryx)
I20260929 16:31:21.238308 30168 webserver.cc:460] Webserver started at http://127.23.44.3:34255/ using document root <none> and password file <none>
I20260929 16:31:21.238857 30168 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:21.238915 30168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:21.239114 30168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:21.241050 30168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "e7511a2044854916bdeb38c7078e48d9"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "41f3224161cd780497964bb4b041ebe3"
server_key_iv: "aa782206ac56af4acdc96d491cab89ae"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.241668 30168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "e7511a2044854916bdeb38c7078e48d9"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "41f3224161cd780497964bb4b041ebe3"
server_key_iv: "aa782206ac56af4acdc96d491cab89ae"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.245345 30168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:31:21.247748 30183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:21.249038 30168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:21.249184 30168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "e7511a2044854916bdeb38c7078e48d9"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "41f3224161cd780497964bb4b041ebe3"
server_key_iv: "aa782206ac56af4acdc96d491cab89ae"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.249325 30168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:21.261619 30168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:21.264503 30168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:21.264648 30168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:21.265164 30168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:21.266021 30168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:21.266067 30168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:21.266106 30168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:21.266121 30168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:21.275015 30168 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:45979
I20260929 16:31:21.275084 30296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:45979 every 8 connection(s)
I20260929 16:31:21.275876 30168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:21.283903 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 30168
I20260929 16:31:21.284044 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:21.284348 23728 external_mini_cluster.cc:1428] Setting key 6bd9086b4be7522ebdbc619e9a6bc1c9
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
I20260929 16:31:21.290107 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.277720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:36971 (local address 127.23.44.62:44913)
0929 16:31:21.277942 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.277947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.278636 (+ 689us) server_negotiation.cc:407] Connection header received
0929 16:31:21.280121 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.280124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.280173 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.280255 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.281503 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.282035 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.283295 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.283516 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.283592 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.286695 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.286709 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.286712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.286741 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.288195 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.288668 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.288670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.288672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.288742 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.289097 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.289100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.289101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.289370 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.289453 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.289899 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.289992 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:31:21.290947 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.278072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.3:36971)
0929 16:31:21.278493 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.278535 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.279901 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.280408 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.280415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.280762 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.281356 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.281370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.282178 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.282181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.283185 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.283194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.283638 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.283646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.283796 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.284488 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.284508 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.286500 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.288309 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.288314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.288325 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.288574 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.288845 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.288848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.288850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.288986 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.289471 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.289476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.289694 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.289860 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.290105 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
I20260929 16:31:21.292009 30297 heartbeater.cc:347] Connected to a master server at 127.23.44.62:44913
I20260929 16:31:21.292215 30297 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:21.292623 30297 heartbeater.cc:511] Master 127.23.44.62:44913 requested a full tablet report, sending...
I20260929 16:31:21.293584 29838 ts_manager.cc:194] Registered new tserver with Master: e7511a2044854916bdeb38c7078e48d9 (127.23.44.3:45979)
I20260929 16:31:21.294163 29838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:36971
I20260929 16:31:21.299446 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.44.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.61.keytab.
Entry for principal oryx/127.23.44.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.61.keytab.
I20260929 16:31:21.328418 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.61:43217
--webserver_interface=127.23.44.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.61.keytab
--principal=oryx/127.23.44.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:44913,127.23.44.61:43217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:21.440653 30302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:21.440913 30302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:21.440953 30302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:21.445046 30302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:21.445103 30302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:21.445122 30302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:21.445138 30302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:21.445176 30302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:21.450400 30302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:44913,127.23.44.61:43217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.61.keytab
--principal=oryx/127.23.44.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.61:43217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.44.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
W20260929 16:31:21.450614 30302 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.44.62:44913,127.23.44.61:43217'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 16:31:21.451453 30302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:21.452495 30302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:21.461021 30310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:21.461172 30308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:21.461021 30307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:21.461354 30302 server_base.cc:1034] running on GCE node
I20260929 16:31:21.461866 30302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:21.462919 30302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:21.464108 30302 hybrid_clock.cc:648] HybridClock initialized: now 1790699481464089 us; error 32 us; skew 500 ppm
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:21.467025 30302 init.cc:377] Logged in from keytab as oryx/127.23.44.61@KRBTEST.COM (short username oryx)
I20260929 16:31:21.467895 30302 webserver.cc:460] Webserver started at http://127.23.44.61:46457/ using document root <none> and password file <none>
I20260929 16:31:21.468427 30302 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:21.468487 30302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:21.468652 30302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:21.470769 30302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data/instance:
uuid: "7ad0e747da51470fbc4a8d4458498b64"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "623b91371a872a0899e350994f8a7562"
server_key_iv: "36b7e25a151dcdd87de58f470728e399"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.471306 30302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/wal/instance:
uuid: "7ad0e747da51470fbc4a8d4458498b64"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "623b91371a872a0899e350994f8a7562"
server_key_iv: "36b7e25a151dcdd87de58f470728e399"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.475185 30302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:31:21.477833 30317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:21.479398 30302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:31:21.479508 30302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/wal
uuid: "7ad0e747da51470fbc4a8d4458498b64"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "623b91371a872a0899e350994f8a7562"
server_key_iv: "36b7e25a151dcdd87de58f470728e399"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.479593 30302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:21.514892 30302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:21.517946 30302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:21.518098 30302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:21.525669 30302 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.61:43217
I20260929 16:31:21.525938 30369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.61:43217 every 8 connection(s)
I20260929 16:31:21.526803 30302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
I20260929 16:31:21.529320 30370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:21.532797 30370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:31:21.534955 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 30302
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.61@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.44.61@KRBTEST.COM
I20260929 16:31:21.547099 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.534617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.23.44.62:44913)
0929 16:31:21.534798 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.534804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.535637 (+ 833us) server_negotiation.cc:407] Connection header received
0929 16:31:21.536712 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.536716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.536813 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.536924 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.537834 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.538372 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.539557 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.539890 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.540009 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.543004 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.543014 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.543016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.543059 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.544803 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.545364 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.545370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.545372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.545459 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.545843 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.545847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.545849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.546076 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.546207 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.546806 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.546930 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:31:21.547845 30372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.534989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:52506)
0929 16:31:21.535490 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.535530 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.536527 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.537005 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.537009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.537457 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.537690 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.537702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.538494 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.538497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.539424 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.539432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.540050 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.540054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.540233 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.540973 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.540988 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.542850 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.544945 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.544954 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.544964 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.545234 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.545561 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.545564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.545566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.545732 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.546201 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.546208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.546393 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.546580 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.546927 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"spinlock_wait_cycles":13952,"thread_start_us":99,"threads_started":1}
I20260929 16:31:21.547847 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.535754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.23.44.61:43217)
0929 16:31:21.536079 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.536090 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.536130 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:31:21.536549 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.536572 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.536805 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.536979 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.537925 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.538956 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.540005 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.540482 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.540587 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.543513 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.543520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.543539 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.543566 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.545132 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.545594 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.545599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.545601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.545710 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.545997 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.546000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.546002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.546417 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.546636 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.546910 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.547045 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"spinlock_wait_cycles":3968,"thread_start_us":158,"threads_started":1}
I20260929 16:31:21.549393 30370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.61@KRBTEST.COM for oryx/127.23.44.61@KRBTEST.COM
I20260929 16:31:21.557941 30372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.549962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:43217 (local address 127.0.0.1:38934)
0929 16:31:21.550060 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.550074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.550147 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.550364 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.550366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.550512 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.550597 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.550602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.551350 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.551353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.551907 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.551916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.552396 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.552400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.552515 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.553038 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.553049 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.554570 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.556562 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.556564 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.556566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.556765 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.557048 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.557050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.557051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.557127 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.557542 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.557544 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.557621 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.557710 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.557816 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 16:31:21.558076 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.550068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.23.44.61:43217)
0929 16:31:21.550178 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.550182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.550203 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:31:21.550234 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.550236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.550269 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.550351 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.550691 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.551224 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.552010 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.552249 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.552405 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.554703 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.554707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.554710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.554736 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.556444 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.556861 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.556865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.556867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.556953 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.557213 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.557216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.557218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.557447 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.557542 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.557783 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.557897 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 16:31:21.561968 30370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: Bootstrap starting.
I20260929 16:31:21.564742 30370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:21.565649 30370 log.cc:824] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:21.567878 30370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: No bootstrap required, opened a new log
I20260929 16:31:21.571177 30370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } }
I20260929 16:31:21.571380 30370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:21.571416 30370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad0e747da51470fbc4a8d4458498b64, State: Initialized, Role: FOLLOWER
I20260929 16:31:21.571789 30370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } }
I20260929 16:31:21.572199 30376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } } }
I20260929 16:31:21.572336 30376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:31:21.572932 30370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:31:21.577782 30387 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:31:21.577864 30387 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:31:21.580212 30302 master_runner.cc:428] Detected that this master 7ad0e747da51470fbc4a8d4458498b64 is joining an existing cluster
I20260929 16:31:21.580281 30302 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.44.61:43217
I20260929 16:31:21.591225 30393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.582613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:52516)
0929 16:31:21.583061 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.583076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.583178 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.583561 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.583564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.583813 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.584003 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.584010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.584961 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.584967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.586000 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.586011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.586607 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.586610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.586677 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.587255 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.587266 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.587663 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.589441 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.589444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.589447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.589680 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.590042 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.590046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.590048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.590142 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.590675 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.590678 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.590770 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.590881 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.591024 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":36,"thread_start_us":92,"threads_started":1}
I20260929 16:31:21.591248 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.582727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.23.44.62:44913)
0929 16:31:21.582861 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.582866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.583157 (+ 291us) server_negotiation.cc:407] Connection header received
0929 16:31:21.583298 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.583313 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.583361 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.583415 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.584238 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.584858 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.586123 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.586474 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.586578 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.587778 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.587783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.587786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.587811 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.589333 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.589839 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.589843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.589846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.589935 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.590328 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.590332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.590344 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.590571 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.590675 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.590952 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.591075 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:31:21.593014 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.583079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:43217 (local address 127.0.0.1:38940)
0929 16:31:21.583376 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.583390 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.583482 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.583706 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.583709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.583909 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.584042 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.584047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.585129 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.585132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.586015 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.586021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.587233 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.587236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.587337 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.588640 (+ 1303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.588655 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.589063 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.591043 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.591047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.591049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.591260 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.592061 (+ 801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.592066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.592068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.592157 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.592509 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.592512 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.592589 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.592691 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.592824 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":49,"thread_start_us":109,"threads_started":1}
I20260929 16:31:21.593001 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.583375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.23.44.61:43217)
0929 16:31:21.583468 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.583472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.583486 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:31:21.583575 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.583579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.583613 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.583697 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.584276 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.585021 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.586687 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.587037 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.587135 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.589213 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.589218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.589221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.589247 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.590937 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.591852 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.591856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.591858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.591940 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.592228 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.592233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.592234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.592412 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.592509 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.592743 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.592873 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":30}
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699480, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.62@KRBTEST.COM for oryx/127.23.44.61@KRBTEST.COM
I20260929 16:31:21.604895 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.595648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.23.44.61:43217)
0929 16:31:21.595774 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.595778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.596125 (+ 347us) server_negotiation.cc:407] Connection header received
0929 16:31:21.596285 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.596287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.596325 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.596388 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.597083 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.597610 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.598793 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.599013 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.599087 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.601602 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.601605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.601607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.601626 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.603259 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.603779 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.603786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.603791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.603875 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.604153 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.604155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.604156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.604297 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.604386 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.604711 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.604791 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:31:21.605018 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.595624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:43217 (local address 127.0.0.1:38942)
0929 16:31:21.595996 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.596042 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.596166 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.596444 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.596450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.596841 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.596969 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.596976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.597710 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.597713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.598669 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.598680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.599114 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.599117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.599230 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.599698 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.599713 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.601460 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.603376 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.603384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.603391 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.603672 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.603968 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.603971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.603973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.604067 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.604393 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.604400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.604520 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.604669 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.604849 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260929 16:31:21.606353 29838 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.44.61:43217: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64"
member_type: NON_VOTER
last_known_addr {
host: "127.23.44.61"
port: 43217
}
attrs {
promote: true
}
}
dest_uuid: "939af390c9f94d3c93c53b23f0754cc2"
cas_config_opid_index: -1
I20260929 16:31:21.607004 29838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true } }
I20260929 16:31:21.608834 30398 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7ad0e747da51470fbc4a8d4458498b64 (127.23.44.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true } } }
I20260929 16:31:21.609679 30344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:21.610739 30398 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.44.61:43217
I20260929 16:31:21.610817 30344 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 FOLLOWER]: Refusing update from remote peer 939af390c9f94d3c93c53b23f0754cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 16:31:21.611097 30302 master.cc:537] Master@127.23.44.61:43217 shutting down...
I20260929 16:31:21.611142 30397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true } } }
I20260929 16:31:21.612323 30398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.23.44.60@KRBTEST.COM; defaulting to no policy
I20260929 16:31:21.614599 30397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:21.615607 30302 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:21.615897 30302 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:21.616017 30302 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.60@KRBTEST.COM" created.
W20260929 16:31:21.617019 29824 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 16:31:21.618460 29824 consensus_peers.cc:487] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 -> Peer 7ad0e747da51470fbc4a8d4458498b64 (127.23.44.61:43217): Couldn't send request to peer 7ad0e747da51470fbc4a8d4458498b64. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 16:31:21.620999 30302 master.cc:552] Master@127.23.44.61:43217 shutdown complete.
I20260929 16:31:21.621047 30302 master_runner.cc:305] Clearing existing system tablet
I20260929 16:31:21.621294 30302 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:21.621357 30302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:21.622581 30302 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:21.623526 30403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:21.623818 30302 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:21.623940 30302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 16:31:21.623973 30302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/wal
uuid: "7ad0e747da51470fbc4a8d4458498b64"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "623b91371a872a0899e350994f8a7562"
server_key_iv: "36b7e25a151dcdd87de58f470728e399"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.625895 30302 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:21.629714 30302 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 16:31:21.629799 30302 log.cc:1196] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 16:31:21.630024 30302 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: Deleting consensus metadata
I20260929 16:31:21.630184 30302 master_runner.cc:315] Copying system tablet from 127.23.44.62:44913
I20260929 16:31:21.631292 30302 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: tablet copy: Beginning tablet copy session from remote peer at address 127.23.44.62:44913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.60.keytab.
Entry for principal oryx/127.23.44.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.60.keytab.
I20260929 16:31:21.634725 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.60:38823
--webserver_interface=127.23.44.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.60.keytab
--principal=oryx/127.23.44.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:44913,127.23.44.61:43217,127.23.44.60:38823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:31:21.640524 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.632077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52528 (local address 127.23.44.62:44913)
0929 16:31:21.632219 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.632224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.632412 (+ 188us) server_negotiation.cc:407] Connection header received
0929 16:31:21.632536 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.632539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.632588 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.632677 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.633145 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.634031 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.635212 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.635670 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.635804 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.637121 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.637125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.637128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.637154 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.638791 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.639225 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.639229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.639231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.639323 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.639644 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.639647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.639649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.639858 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.639959 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.640245 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.640366 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:31:21.640832 30409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.632006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:52528)
0929 16:31:21.632325 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.632338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.632443 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.632739 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.632745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.632933 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.633032 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.633038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.634162 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.634165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.635086 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.635096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.635796 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.635800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.635887 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.636533 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.636548 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.636947 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.638907 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.638910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.638912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.639129 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.639443 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.639446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.639448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.639549 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.640000 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.640003 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.640085 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.640176 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.640287 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
I20260929 16:31:21.641408 29858 tablet_copy_service.cc:140] P 939af390c9f94d3c93c53b23f0754cc2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7ad0e747da51470fbc4a8d4458498b64 ({username='oryx', principal='oryx/127.23.44.61@KRBTEST.COM'} at 127.0.0.1:52528)
I20260929 16:31:21.641570 29858 tablet_copy_service.cc:161] P 939af390c9f94d3c93c53b23f0754cc2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7ad0e747da51470fbc4a8d4458498b64 at {username='oryx', principal='oryx/127.23.44.61@KRBTEST.COM'} at 127.0.0.1:52528: session id = 7ad0e747da51470fbc4a8d4458498b64-00000000000000000000000000000000
I20260929 16:31:21.642891 29858 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:31:21.644520 30302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 16:31:21.646785 30302 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: tablet copy: Starting download of 0 data blocks...
I20260929 16:31:21.646968 30302 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: tablet copy: Starting download of 1 WAL segments...
I20260929 16:31:21.648713 30302 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:31:21.650908 29858 tablet_copy_service.cc:342] P 939af390c9f94d3c93c53b23f0754cc2: Request end of tablet copy session 7ad0e747da51470fbc4a8d4458498b64-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.44.61@KRBTEST.COM'} at 127.0.0.1:52528
I20260929 16:31:21.651029 29858 tablet_copy_service.cc:433] P 939af390c9f94d3c93c53b23f0754cc2: ending tablet copy session 7ad0e747da51470fbc4a8d4458498b64-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7ad0e747da51470fbc4a8d4458498b64
I20260929 16:31:21.658600 30302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:21.660391 30412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:21.660465 30411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:21.662264 30414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:21.662393 30302 server_base.cc:1034] running on GCE node
I20260929 16:31:21.662657 30302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:21.662940 30302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:21.664075 30302 hybrid_clock.cc:648] HybridClock initialized: now 1790699481664063 us; error 29 us; skew 500 ppm
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:21.665706 30302 init.cc:377] Logged in from keytab as oryx/127.23.44.61@KRBTEST.COM (short username oryx)
I20260929 16:31:21.666152 30302 webserver.cc:460] Webserver started at http://127.23.44.61:40191/ using document root <none> and password file <none>
I20260929 16:31:21.666358 30302 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:21.666400 30302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:21.667695 30302 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:21.668659 30421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:21.668891 30302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:21.668960 30302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/wal
uuid: "7ad0e747da51470fbc4a8d4458498b64"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "623b91371a872a0899e350994f8a7562"
server_key_iv: "36b7e25a151dcdd87de58f470728e399"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.669028 30302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:21.681372 30302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:21.683387 30302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:21.688064 30302 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.61:43217
I20260929 16:31:21.688612 30302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
I20260929 16:31:21.689482 30473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.61:43217 every 8 connection(s)
I20260929 16:31:21.690425 30474 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:31:21.691383 30474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: Bootstrap starting.
I20260929 16:31:21.701190 30474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:31:21.701534 30474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: Bootstrap complete.
I20260929 16:31:21.702078 30474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true } }
I20260929 16:31:21.702225 30474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ad0e747da51470fbc4a8d4458498b64, State: Initialized, Role: LEARNER
I20260929 16:31:21.702374 30474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true } }
I20260929 16:31:21.702754 30477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true } } }
I20260929 16:31:21.702890 30474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:21.702980 30477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:31:21.706736 30488 catalog_manager.cc:1259] Loaded cluster ID: e33f563b1bbe46d5a0c2b73bb61906b7
I20260929 16:31:21.706791 30488 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: loading cluster ID for follower catalog manager: success
I20260929 16:31:21.708747 30488 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: acquiring CA information for follower catalog manager: success
I20260929 16:31:21.710062 30488 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:31:21.770875 30410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:21.771112 30410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:21.771183 30410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:21.775180 30410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:21.775236 30410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:21.775256 30410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:21.775272 30410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:21.775287 30410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:21.780766 30410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:44913,127.23.44.61:43217,127.23.44.60:38823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.60.keytab
--principal=oryx/127.23.44.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.60:38823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.44.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:21.781867 30410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:21.782882 30410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:21.791312 30496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:21.791368 30493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:21.791512 30410 server_base.cc:1034] running on GCE node
W20260929 16:31:21.791605 30494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:21.792165 30410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:21.793329 30410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:21.794557 30410 hybrid_clock.cc:648] HybridClock initialized: now 1790699481794543 us; error 30 us; skew 500 ppm
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:21.797492 30410 init.cc:377] Logged in from keytab as oryx/127.23.44.60@KRBTEST.COM (short username oryx)
I20260929 16:31:21.798475 30410 webserver.cc:460] Webserver started at http://127.23.44.60:40351/ using document root <none> and password file <none>
I20260929 16:31:21.799023 30410 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:21.799114 30410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:21.799337 30410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:21.801399 30410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data/instance:
uuid: "09d573aa99e74de599580ced2ce89990"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "f2a976e16d0f241d7d35300b88a0fddb"
server_key_iv: "d12c2a9522d955599600a5e213245313"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.801935 30410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/wal/instance:
uuid: "09d573aa99e74de599580ced2ce89990"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "f2a976e16d0f241d7d35300b88a0fddb"
server_key_iv: "d12c2a9522d955599600a5e213245313"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.805794 30410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:31:21.808310 30503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:21.809569 30410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:21.809706 30410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/wal
uuid: "09d573aa99e74de599580ced2ce89990"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "f2a976e16d0f241d7d35300b88a0fddb"
server_key_iv: "d12c2a9522d955599600a5e213245313"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.809813 30410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:21.823079 30410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:21.826624 30410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:21.826829 30410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:21.833521 30410 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.60:38823
I20260929 16:31:21.833832 30555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.60:38823 every 8 connection(s)
I20260929 16:31:21.834465 30410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
I20260929 16:31:21.837095 30556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:21.837797 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 30410
I20260929 16:31:21.842036 30556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.44.60@KRBTEST.COM
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.60@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
I20260929 16:31:21.852445 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.839315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52272 (local address 127.23.44.60:38823)
0929 16:31:21.839725 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.839739 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.839778 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:31:21.840451 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.840467 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.840774 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.841044 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.841979 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.843027 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.843998 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.844624 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.844792 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.847356 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.847362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.847370 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.847394 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.849441 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.849946 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.849952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.849957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.850077 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.850411 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.850417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.850418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.850913 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.851111 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.851397 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.851671 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260929 16:31:21.853098 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.843787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.23.44.62:44913)
0929 16:31:21.843934 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.843941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.844156 (+ 215us) server_negotiation.cc:407] Connection header received
0929 16:31:21.844277 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.844280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.844333 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.844418 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.845047 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.845605 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.846658 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.847028 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.847152 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.849759 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.849762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.849764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.849786 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.851263 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.851723 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.851728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.851730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.851814 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.852104 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.852109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.852111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.852349 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.852453 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.852823 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.852943 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:31:21.853140 30559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.843647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:52540)
0929 16:31:21.844026 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.844071 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.844186 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.844472 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.844476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.844783 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.844917 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.844923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.845732 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.845738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.846532 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.846542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.847187 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.847192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.847301 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.847845 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.847860 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.849627 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.851380 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.851390 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.851395 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.851617 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.851915 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.851918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.851920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.852015 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.852452 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.852459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.852595 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.852749 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.852944 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":1,"thread_start_us":157,"threads_started":1}
I20260929 16:31:21.854270 30556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.60@KRBTEST.COM for oryx/127.23.44.61@KRBTEST.COM
I20260929 16:31:21.862280 30559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.854730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:43217 (local address 127.0.0.1:38954)
0929 16:31:21.854847 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.854861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.854928 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.855362 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.855365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.855523 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.855603 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.855608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.856502 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.856506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.857032 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.857039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.857551 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.857555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.857650 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.858094 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.858107 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.859219 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.860922 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.860924 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.860926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.861106 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.861422 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.861424 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.861426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.861499 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.861914 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.861916 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.861988 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.862071 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.862164 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 16:31:21.862349 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.854891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.23.44.61:43217)
0929 16:31:21.855163 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.855167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.855178 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:31:21.855211 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.855213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.855249 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.855350 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.855714 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.856380 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.857145 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.857441 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.857555 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.859345 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.859348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.859350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.859371 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.860797 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.861194 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.861197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.861199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.861339 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.861617 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.861620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.861621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.861801 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.861921 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.862129 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.862223 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260929 16:31:21.863094 30556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.60@KRBTEST.COM for oryx/127.23.44.60@KRBTEST.COM
I20260929 16:31:21.871140 30559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.863522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:38823 (local address 127.0.0.1:52288)
0929 16:31:21.863625 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.863638 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.863708 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.863964 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.863966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.864083 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.864146 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.864150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.865221 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.865224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.865839 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.865848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.866297 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.866300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.866404 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.866840 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.866851 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.868038 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.869850 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.869853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.869854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.870044 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.870321 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.870326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.870331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.870431 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.870766 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.870768 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.870839 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.870927 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.871021 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 16:31:21.871169 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.863684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.23.44.60:38823)
0929 16:31:21.863806 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.863810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.863828 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:31:21.863859 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.863861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.863891 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.863948 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.864261 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.864740 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.865958 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.866180 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.866262 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.868156 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.868159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.868161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.868180 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.869738 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.870130 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.870133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.870134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.870201 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.870523 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.870525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.870526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.870683 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.870758 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.870983 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.871067 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:31:21.872037 30029 heartbeater.cc:503] Master 127.23.44.62:44913 was elected leader, sending a full tablet report...
I20260929 16:31:21.874404 30556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: Bootstrap starting.
I20260929 16:31:21.876610 30556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:21.877391 30556 log.cc:824] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:21.879165 30556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: No bootstrap required, opened a new log
I20260929 16:31:21.882318 30556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } }
I20260929 16:31:21.882547 30556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:21.882629 30556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d573aa99e74de599580ced2ce89990, State: Initialized, Role: FOLLOWER
I20260929 16:31:21.883095 30556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } }
I20260929 16:31:21.883579 30562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } } }
I20260929 16:31:21.883740 30562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:31:21.884276 30556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:31:21.889117 30573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:31:21.889197 30573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:31:21.892967 30410 master_runner.cc:428] Detected that this master 09d573aa99e74de599580ced2ce89990 is joining an existing cluster
I20260929 16:31:21.893033 30410 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.44.60:38823
I20260929 16:31:21.904557 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.895721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:43217 (local address 127.0.0.1:38970)
0929 16:31:21.896345 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.896359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.896458 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.896747 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.896749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.896978 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.897119 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.897124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.898118 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.898130 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.898822 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.898832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.899592 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.899595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.899836 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.900407 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.900422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.900861 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.903135 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.903139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.903142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.903373 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.903647 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.903649 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.903650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.903726 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.904128 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.904130 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.904233 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.904318 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.904413 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":56,"spinlock_wait_cycles":1408,"thread_start_us":73,"threads_started":1}
I20260929 16:31:21.904667 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.896390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.23.44.61:43217)
0929 16:31:21.896518 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.896522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.896536 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:31:21.896573 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.896576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.896624 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.896708 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.897250 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.898009 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.898980 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.899336 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.899412 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.900993 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.900998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.901001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.901026 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.902768 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.903462 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.903466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.903468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.903554 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.903811 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.903814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.903816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.904033 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.904131 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.904393 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.904515 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:31:21.905790 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 29900
I20260929 16:31:21.911170 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:36317
--local_ip_for_outbound_sockets=127.23.44.1
--tserver_master_addrs=127.23.44.62:44913
--webserver_port=41531
--webserver_interface=127.23.44.1
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab
--principal=oryx/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.44.62:44913,127.23.44.61:43217
--tserver_master_addrs=127.23.44.62:44913,127.23.44.61:43217,127.23.44.60:38823 with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:31:21.920794 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.896085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.23.44.60:38823)
0929 16:31:21.896189 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.896193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.896980 (+ 787us) server_negotiation.cc:407] Connection header received
0929 16:31:21.897123 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.897126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.897166 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.897230 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.897750 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.898513 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.899977 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.900667 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.900742 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.902175 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.902177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.902179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.902201 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.918500 (+ 16299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.919042 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.919046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.919047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.919120 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.919385 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.919388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.919390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.919859 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.919929 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.920517 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.920633 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":35}
I20260929 16:31:21.921476 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.896075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:38823 (local address 127.0.0.1:52290)
0929 16:31:21.896906 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.896920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.897034 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.897364 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.897366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.897542 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.897642 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.897648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.898631 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.898634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.899856 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.899865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.900872 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.900875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.900978 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.901461 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.901479 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.901843 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.918631 (+ 16788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.918634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.918637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.918945 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.919210 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.919213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.919215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.919301 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.920248 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.920251 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.920352 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.921134 (+ 782us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.921305 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":84,"spinlock_wait_cycles":4736,"thread_start_us":80,"threads_started":1}
I20260929 16:31:21.923144 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.895360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.23.44.62:44913)
0929 16:31:21.895484 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.895489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.895669 (+ 180us) server_negotiation.cc:407] Connection header received
0929 16:31:21.895867 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.895871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.895924 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.896009 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.896648 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.897513 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.899579 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.899932 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.900006 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.919127 (+ 19121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.919134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.919139 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.919178 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.921001 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.921652 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.921659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.921663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.921752 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.922060 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.922063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.922065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.922262 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.922326 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.922882 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.922995 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:31:21.923830 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.895275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:52554)
0929 16:31:21.895567 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.895585 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.895743 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.896131 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.896134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.896357 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.896487 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.896495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.897652 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.897656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.899439 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.899449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.900153 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.900156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.900241 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.917832 (+ 17591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.917855 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.918918 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.921210 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.921213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.921215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.921518 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.921876 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.921879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.921881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.921962 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.922610 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.922613 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.922696 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.923530 (+ 834us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.923664 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":30,"thread_start_us":125,"threads_started":1}
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699480, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.62@KRBTEST.COM for oryx/127.23.44.60@KRBTEST.COM
I20260929 16:31:21.937827 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.927616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:38823 (local address 127.0.0.1:52304)
0929 16:31:21.927831 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.927847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.927947 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.928228 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.928232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.928473 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.928581 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.928587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.929605 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.929609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.930583 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.930593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.931146 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.931150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.931263 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.931728 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.931743 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.933350 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.936146 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.936149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.936152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.936402 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.936704 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.936707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.936709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.936799 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.937323 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.937326 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.937420 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.937528 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.937661 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260929 16:31:21.937855 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.927702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.23.44.60:38823)
0929 16:31:21.927805 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.927809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.927914 (+ 105us) server_negotiation.cc:407] Connection header received
0929 16:31:21.928061 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.928064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.928114 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.928196 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.928701 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.929486 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.930705 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.931031 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.931130 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.933501 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.933506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.933509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.933534 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.936034 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.936505 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.936508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.936510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.936600 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.936949 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.936952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.936954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.937177 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.937302 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.937595 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.937710 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 16:31:21.939059 29838 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.44.60:38823: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "09d573aa99e74de599580ced2ce89990"
member_type: NON_VOTER
last_known_addr {
host: "127.23.44.60"
port: 38823
}
attrs {
promote: true
}
}
dest_uuid: "939af390c9f94d3c93c53b23f0754cc2"
cas_config_opid_index: 5
I20260929 16:31:21.939663 29838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } }
I20260929 16:31:21.940717 30398 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 09d573aa99e74de599580ced2ce89990 (127.23.44.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:21.941753 30530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:21.942205 30398 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.44.60:38823
I20260929 16:31:21.942374 30398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:21.942467 30398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:21.943015 30410 master.cc:537] Master@127.23.44.60:38823 shutting down...
I20260929 16:31:21.943022 30530 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 1 FOLLOWER]: Refusing update from remote peer 939af390c9f94d3c93c53b23f0754cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 16:31:21.944756 30397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:31:21.947009 29824 consensus_peers.cc:487] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 -> Peer 09d573aa99e74de599580ced2ce89990 (127.23.44.60:38823): Couldn't send request to peer 09d573aa99e74de599580ced2ce89990. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 16:31:21.947976 30410 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:21.948240 30410 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:21.948329 30410 tablet_replica.cc:330] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: stopping tablet replica
I20260929 16:31:21.951376 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.941106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.23.44.61:43217)
0929 16:31:21.941214 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.941219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.946616 (+ 5397us) server_negotiation.cc:407] Connection header received
0929 16:31:21.946686 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.946705 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.946774 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.946865 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:21.947429 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.948194 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.949810 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.950451 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.950528 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.950655 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.951166 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.951241 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:31:21.951731 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.940897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:43217 (local address 127.0.0.1:38980)
0929 16:31:21.946489 (+ 5592us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.946507 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.946601 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.947173 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.947177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.947194 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:21.947276 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.947296 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.948344 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.948348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.949669 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.949679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.950918 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.950922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.951013 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.951054 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.951552 (+ 498us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.951624 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2257}
I20260929 16:31:21.952318 30448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 LEARNER]: Refusing update from remote peer 939af390c9f94d3c93c53b23f0754cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 16:31:21.952723 30397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 16:31:21.954526 30448 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 09d573aa99e74de599580ced2ce89990 (127.23.44.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:21.956434 30477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 939af390c9f94d3c93c53b23f0754cc2. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:21.956539 30477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:31:21.956614 30477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: NON_VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: true } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:21.956676 30477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:31:21.957180 30397 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2: attempting to promote NON_VOTER 7ad0e747da51470fbc4a8d4458498b64 to VOTER
I20260929 16:31:21.957195 30410 master.cc:552] Master@127.23.44.60:38823 shutdown complete.
I20260929 16:31:21.957237 30410 master_runner.cc:305] Clearing existing system tablet
I20260929 16:31:21.957528 30410 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:21.957580 30410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:21.957607 30397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } }
I20260929 16:31:21.959052 30410 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:21.959115 30448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 LEARNER]: Refusing update from remote peer 939af390c9f94d3c93c53b23f0754cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 16:31:21.959462 30398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 16:31:21.960017 30587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:21.960253 30410 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:21.960386 30410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:21.960427 30410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/wal
uuid: "09d573aa99e74de599580ced2ce89990"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "f2a976e16d0f241d7d35300b88a0fddb"
server_key_iv: "d12c2a9522d955599600a5e213245313"
server_key_version: "encryptionkey@0"
I20260929 16:31:21.960906 30398 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7ad0e747da51470fbc4a8d4458498b64 (127.23.44.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:21.962199 30398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:21.962304 30398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:21.962404 30398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:21.962468 30398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:21.962499 30410 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:21.963129 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.958755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:38823 (local address 127.0.0.1:52306)
0929 16:31:21.962952 (+ 4197us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.963020 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:38823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4127}
I20260929 16:31:21.963894 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.963551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:38823 (local address 127.0.0.1:52314)
0929 16:31:21.963782 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.963827 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:38823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
W20260929 16:31:21.963961 29824 consensus_peers.cc:487] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 -> Peer 09d573aa99e74de599580ced2ce89990 (127.23.44.60:38823): Couldn't send request to peer 09d573aa99e74de599580ced2ce89990. Status: Network error: Client connection negotiation failed: client connection to 127.23.44.60:38823: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:31:21.964854 30410 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 16:31:21.964947 30410 log.cc:1196] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 16:31:21.965194 30410 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: Deleting consensus metadata
I20260929 16:31:21.965751 30447 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7ad0e747da51470fbc4a8d4458498b64 (127.23.44.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:21.968093 30410 master_runner.cc:315] Copying system tablet from 127.23.44.62:44913
I20260929 16:31:21.969365 30410 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: tablet copy: Beginning tablet copy session from remote peer at address 127.23.44.62:44913
I20260929 16:31:21.969726 30584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:21.969857 30584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:31:21.982462 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.970240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.23.44.62:44913)
0929 16:31:21.970514 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:21.970519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:21.973526 (+ 3007us) server_negotiation.cc:407] Connection header received
0929 16:31:21.973713 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:21.973716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:21.973765 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:21.973839 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:21.974460 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.975279 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.976518 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.976970 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.977056 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.978573 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:21.978577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:21.978579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:21.978604 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:21.980336 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.981097 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.981101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.981103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.981189 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:21.981537 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:21.981540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:21.981542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:21.981759 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:21.981856 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:21.982185 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 16:31:21.982307 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260929 16:31:21.982481 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:21.969874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:52566)
0929 16:31:21.973423 (+ 3549us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:21.973436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:21.973587 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:21.973962 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:21.973965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:21.974182 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:21.974312 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.974319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.975423 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.975426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.976374 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:21.976384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:21.977249 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:21.977289 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:21.977393 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:21.977971 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:21.977985 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:21.978419 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:21.980594 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.980598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.980601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.980990 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.981330 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:21.981334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:21.981336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:21.981436 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:21.981869 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:21.981872 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:21.981987 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 16:31:21.982105 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:31:21.982253 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3472,"thread_start_us":93,"threads_started":1}
I20260929 16:31:21.983292 29858 tablet_copy_service.cc:140] P 939af390c9f94d3c93c53b23f0754cc2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 09d573aa99e74de599580ced2ce89990 ({username='oryx', principal='oryx/127.23.44.60@KRBTEST.COM'} at 127.0.0.1:52566)
I20260929 16:31:21.983384 29858 tablet_copy_service.cc:161] P 939af390c9f94d3c93c53b23f0754cc2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 09d573aa99e74de599580ced2ce89990 at {username='oryx', principal='oryx/127.23.44.60@KRBTEST.COM'} at 127.0.0.1:52566: session id = 09d573aa99e74de599580ced2ce89990-00000000000000000000000000000000
I20260929 16:31:21.984437 29858 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:31:21.985844 30410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 16:31:21.988428 30410 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: tablet copy: Starting download of 0 data blocks...
I20260929 16:31:21.988617 30410 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: tablet copy: Starting download of 1 WAL segments...
I20260929 16:31:21.990114 30410 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:31:21.991885 29858 tablet_copy_service.cc:342] P 939af390c9f94d3c93c53b23f0754cc2: Request end of tablet copy session 09d573aa99e74de599580ced2ce89990-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.44.60@KRBTEST.COM'} at 127.0.0.1:52566
I20260929 16:31:21.991962 29858 tablet_copy_service.cc:433] P 939af390c9f94d3c93c53b23f0754cc2: ending tablet copy session 09d573aa99e74de599580ced2ce89990-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 09d573aa99e74de599580ced2ce89990
I20260929 16:31:21.998718 30410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:22.000443 30595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:22.000535 30598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:22.000730 30410 server_base.cc:1034] running on GCE node
W20260929 16:31:22.000779 30596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:22.000988 30410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:22.001284 30410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:22.002408 30410 hybrid_clock.cc:648] HybridClock initialized: now 1790699482002365 us; error 57 us; skew 500 ppm
Sep 29 16:31:21 dist-test-slave-mlmz krb5kdc[29793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699481, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:22.004055 30410 init.cc:377] Logged in from keytab as oryx/127.23.44.60@KRBTEST.COM (short username oryx)
I20260929 16:31:22.004489 30410 webserver.cc:460] Webserver started at http://127.23.44.60:42681/ using document root <none> and password file <none>
I20260929 16:31:22.004665 30410 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:22.004704 30410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:22.005976 30410 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 16:31:22.006947 30605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:22.007184 30410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 16:31:22.007285 30410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/wal
uuid: "09d573aa99e74de599580ced2ce89990"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "f2a976e16d0f241d7d35300b88a0fddb"
server_key_iv: "d12c2a9522d955599600a5e213245313"
server_key_version: "encryptionkey@0"
I20260929 16:31:22.007419 30410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:22.018548 30410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:22.020824 30410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:22.025696 30410 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.60:38823
I20260929 16:31:22.025820 30657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.60:38823 every 8 connection(s)
I20260929 16:31:22.026432 30410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
I20260929 16:31:22.028347 30658 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:31:22.029744 30658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: Bootstrap starting.
I20260929 16:31:22.041684 30658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:31:22.042032 30658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: Bootstrap complete.
I20260929 16:31:22.042644 30658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } }
I20260929 16:31:22.042793 30658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09d573aa99e74de599580ced2ce89990, State: Initialized, Role: LEARNER
I20260929 16:31:22.042925 30658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } }
I20260929 16:31:22.043331 30661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:22.043442 30658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:22.043550 30661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:31:22.046639 30672 catalog_manager.cc:1259] Loaded cluster ID: e33f563b1bbe46d5a0c2b73bb61906b7
I20260929 16:31:22.046782 30672 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: loading cluster ID for follower catalog manager: success
I20260929 16:31:22.049275 30672 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: acquiring CA information for follower catalog manager: success
I20260929 16:31:22.050813 30672 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:31:22.061196 30582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:22.061606 30582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:22.061673 30582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:22.066008 30582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:22.066073 30582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:22.066169 30582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:22.073563 30582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab
--principal=oryx/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:36317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=41531
--tserver_master_addrs=127.23.44.62:44913,127.23.44.61:43217,127.23.44.60:38823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:22.074710 30582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:22.075762 30582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:31:22.078667 30163 heartbeater.cc:503] Master 127.23.44.62:44913 was elected leader, sending a full tablet report...
W20260929 16:31:22.086867 30680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:22.086853 30677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:22.087173 30678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:22.087308 30582 server_base.cc:1034] running on GCE node
I20260929 16:31:22.087669 30582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:22.088274 30582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:22.089440 30582 hybrid_clock.cc:648] HybridClock initialized: now 1790699482089421 us; error 32 us; skew 500 ppm
Sep 29 16:31:22 dist-test-slave-mlmz krb5kdc[29793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699482, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:22.092365 30582 init.cc:377] Logged in from keytab as oryx/127.23.44.1@KRBTEST.COM (short username oryx)
I20260929 16:31:22.093396 30582 webserver.cc:460] Webserver started at http://127.23.44.1:41531/ using document root <none> and password file <none>
I20260929 16:31:22.093948 30582 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:22.094019 30582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:22.097837 30582 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:31:22.099893 30687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:22.101158 30582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:22.101329 30582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "30dfcbab3ef543848c69b4734509af52"
format_stamp: "Formatted at 2026-09-29 16:31:20 on dist-test-slave-mlmz"
server_key: "e10f3625a5222b8efefa1d683e5c92c4"
server_key_iv: "9e4d215ae8a393f0964a3c2749952538"
server_key_version: "encryptionkey@0"
I20260929 16:31:22.101862 30582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:22.130141 30582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:22.133136 30582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:22.133384 30582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:22.133975 30582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:22.134812 30582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:22.134883 30582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:22.134948 30582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:22.134999 30582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:22.145087 30582 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:36317
I20260929 16:31:22.145166 30800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:36317 every 8 connection(s)
I20260929 16:31:22.146109 30582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:22.148826 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 30582
I20260929 16:31:22.149037 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 30034
I20260929 16:31:22.156931 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:38115
--local_ip_for_outbound_sockets=127.23.44.2
--tserver_master_addrs=127.23.44.62:44913
--webserver_port=44373
--webserver_interface=127.23.44.2
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab
--principal=oryx/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.44.62:44913,127.23.44.61:43217
--tserver_master_addrs=127.23.44.62:44913,127.23.44.61:43217,127.23.44.60:38823 with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:31:22 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699482, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
Sep 29 16:31:22 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699482, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.60@KRBTEST.COM
Sep 29 16:31:22 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699482, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.61@KRBTEST.COM
I20260929 16:31:22.165205 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.149921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:38253 (local address 127.23.44.60:38823)
0929 16:31:22.152581 (+ 2660us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.152594 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.152607 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:31:22.152661 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.152664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.152710 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.152793 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.153725 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.154878 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.156144 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.156511 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.156610 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.159857 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.159860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.159862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.159888 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.162542 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.163289 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.163292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.163294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.163385 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.163804 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.163809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.163812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.164055 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.164136 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.164911 (+ 775us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.165035 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2569,"thread_start_us":84,"threads_started":1}
I20260929 16:31:22.165202 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.147943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:41479 (local address 127.23.44.62:44913)
0929 16:31:22.148087 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.148090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.149468 (+ 1378us) server_negotiation.cc:407] Connection header received
0929 16:31:22.150679 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.150686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.150756 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.150921 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.152579 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.153436 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.155078 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.155407 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.155496 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.159459 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.159463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.159466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.159492 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.163015 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.163513 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.163517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.163518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.163596 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.163892 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.163896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.163897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.164096 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.164162 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.164911 (+ 749us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.165035 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:31:22.165433 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.150934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:57109 (local address 127.23.44.61:43217)
0929 16:31:22.151117 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.151123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.152351 (+ 1228us) server_negotiation.cc:407] Connection header received
0929 16:31:22.152471 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.152477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.152540 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.152619 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.153715 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.154615 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.156224 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.156935 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.157016 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.160367 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.160374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.160380 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.160413 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.162677 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.163373 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.163379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.163383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.163483 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.163852 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.163858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.163861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.164118 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.164202 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.164927 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.165070 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:31:22.167008 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.149817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:38823 (local address 127.23.44.1:38253)
0929 16:31:22.150401 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.150414 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.150502 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.152911 (+ 2409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.152914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.153175 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:22.153596 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.153603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.155009 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.155015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.156014 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.156023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.156730 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.156736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.156841 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.157323 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:22.157353 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:22.159700 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:22.162687 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.162695 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.162722 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.163175 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.163501 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.163504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.163506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.163682 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.164223 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:22.164230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:22.164614 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.165393 (+ 779us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.165806 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":362,"spinlock_wait_cycles":20224,"thread_start_us":92,"threads_started":1}
I20260929 16:31:22.167008 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.150175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:43217 (local address 127.23.44.1:57109)
0929 16:31:22.152247 (+ 2072us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.152263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.152363 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.152752 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.152757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.153033 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:22.153562 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.153575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.154765 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.154771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.155955 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.155968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.157869 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.157874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.157980 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.158479 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:22.158493 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:22.160214 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:22.162831 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.162837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.162843 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.163245 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.163613 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.163619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.163622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.163733 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.164289 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:22.164294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:22.164617 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.165640 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.165886 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1990,"mutex_wait_us":201,"spinlock_wait_cycles":12672,"thread_start_us":79,"threads_started":1}
I20260929 16:31:22.167727 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.148222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.1:41479)
0929 16:31:22.149326 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.149380 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.150455 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.151215 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.151226 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.151768 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:22.152425 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.152445 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.153603 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.153607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.154956 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.154966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.155677 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.155680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.155992 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.156671 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:22.156690 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:22.159292 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:22.163142 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.163145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.163147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.163402 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.163705 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.163707 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.163709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.163793 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.164249 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:22.164252 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:22.164614 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.165910 (+ 1296us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.166024 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":110,"spinlock_wait_cycles":67072,"thread_start_us":701,"threads_started":1}
I20260929 16:31:22.168577 30803 heartbeater.cc:347] Connected to a master server at 127.23.44.61:43217
I20260929 16:31:22.168825 30803 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:22.168988 30801 heartbeater.cc:347] Connected to a master server at 127.23.44.60:38823
I20260929 16:31:22.169107 30801 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:22.169333 30802 heartbeater.cc:347] Connected to a master server at 127.23.44.62:44913
I20260929 16:31:22.169413 30802 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:22.169490 30803 heartbeater.cc:511] Master 127.23.44.61:43217 requested a full tablet report, sending...
I20260929 16:31:22.169546 30802 heartbeater.cc:511] Master 127.23.44.62:44913 requested a full tablet report, sending...
I20260929 16:31:22.169494 30801 heartbeater.cc:511] Master 127.23.44.60:38823 requested a full tablet report, sending...
I20260929 16:31:22.170799 29838 ts_manager.cc:194] Re-registered known tserver with Master: 30dfcbab3ef543848c69b4734509af52 (127.23.44.1:36317)
I20260929 16:31:22.171066 30430 ts_manager.cc:194] Registered new tserver with Master: 30dfcbab3ef543848c69b4734509af52 (127.23.44.1:36317)
I20260929 16:31:22.171154 30622 ts_manager.cc:194] Registered new tserver with Master: 30dfcbab3ef543848c69b4734509af52 (127.23.44.1:36317)
I20260929 16:31:22.171792 29838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:41479
W20260929 16:31:22.278267 30809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:22.278545 30809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:22.278708 30809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:22.283417 30809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:22.283504 30809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:22.283592 30809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:22.288932 30809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab
--principal=oryx/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:38115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=44373
--tserver_master_addrs=127.23.44.62:44913,127.23.44.61:43217,127.23.44.60:38823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:22.290127 30809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:22.291096 30809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:31:22.296100 30297 heartbeater.cc:503] Master 127.23.44.62:44913 was elected leader, sending a full tablet report...
W20260929 16:31:22.300642 30814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:22.300649 30815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:22.300808 30809 server_base.cc:1034] running on GCE node
W20260929 16:31:22.300988 30817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:22.301507 30809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:22.302191 30809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:22.305559 30809 hybrid_clock.cc:648] HybridClock initialized: now 1790699482305522 us; error 71 us; skew 500 ppm
Sep 29 16:31:22 dist-test-slave-mlmz krb5kdc[29793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699482, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:22.309031 30809 init.cc:377] Logged in from keytab as oryx/127.23.44.2@KRBTEST.COM (short username oryx)
I20260929 16:31:22.310130 30809 webserver.cc:460] Webserver started at http://127.23.44.2:44373/ using document root <none> and password file <none>
I20260929 16:31:22.310760 30809 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:22.310884 30809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:22.315851 30809 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 16:31:22.318300 30824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:22.320405 30809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 16:31:22.320523 30809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "ba8e8b9ff7ccc9757baf5d133d0d27e1"
server_key_iv: "4e89a42190afa75289915a0f91bd8a31"
server_key_version: "encryptionkey@0"
I20260929 16:31:22.321105 30809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:22.349540 30809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:22.354149 30809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:22.354338 30809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:22.354946 30809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:22.355811 30809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:22.355857 30809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:22.355897 30809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:22.355912 30809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:22.365892 30809 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:38115
I20260929 16:31:22.365911 30937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:38115 every 8 connection(s)
I20260929 16:31:22.366886 30809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:22.375458 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 30809
I20260929 16:31:22.375648 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 30168
Sep 29 16:31:22 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699482, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
Sep 29 16:31:22 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699482, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.60@KRBTEST.COM
Sep 29 16:31:22 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699482, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.61@KRBTEST.COM
I20260929 16:31:22.385020 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:45979
--local_ip_for_outbound_sockets=127.23.44.3
--tserver_master_addrs=127.23.44.62:44913
--webserver_port=34255
--webserver_interface=127.23.44.3
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab
--principal=oryx/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.44.62:44913,127.23.44.61:43217
--tserver_master_addrs=127.23.44.62:44913,127.23.44.61:43217,127.23.44.60:38823 with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:31:22.387324 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.370826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:38489 (local address 127.23.44.61:43217)
0929 16:31:22.370960 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.370964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.371077 (+ 113us) server_negotiation.cc:407] Connection header received
0929 16:31:22.371906 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.371918 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.371973 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.372056 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.375060 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.376026 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.380117 (+ 4091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.380584 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.380729 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.383378 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.383387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.383394 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.383432 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.385435 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.385954 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.385961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.385966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.386054 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.386418 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.386424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.386428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.386635 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.386713 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.387041 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.387163 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:31:22.387354 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.371192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:41793 (local address 127.23.44.60:38823)
0929 16:31:22.371436 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.371441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.371461 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:31:22.371710 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.371716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.371782 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.371868 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.374035 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.374842 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.376921 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.377427 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.377713 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.381538 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.381542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.381545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.381585 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.384291 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.385060 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.385064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.385066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.385161 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.385892 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.385896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.385897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.386176 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.386249 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.387050 (+ 801us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.387195 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:31:22.387354 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.368804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:58953 (local address 127.23.44.62:44913)
0929 16:31:22.368969 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.368974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.370430 (+ 1456us) server_negotiation.cc:407] Connection header received
0929 16:31:22.371598 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.371601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.371867 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.371940 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.374036 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.374641 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.376882 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.377384 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.377544 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.381530 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.381534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.381537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.381569 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.384802 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.385905 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.385908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.385910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.386048 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.386417 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.386420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.386422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.386624 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.386730 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.387075 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.387192 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:31:22.388280 30943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.370717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:43217 (local address 127.23.44.2:38489)
0929 16:31:22.370994 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.371009 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.371405 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.372082 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.372087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.372715 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:22.373751 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.373775 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.378939 (+ 5164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.378945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.379924 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.379938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.380897 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.380903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.381037 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.381619 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:22.381638 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:22.383234 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:22.385560 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.385566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.385571 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.385839 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.386161 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.386166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.386171 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.386289 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.386761 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:22.386763 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:22.386840 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.386990 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.387338 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":88,"spinlock_wait_cycles":19072,"thread_start_us":104,"threads_started":1}
I20260929 16:31:22.388564 30944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.371070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:38823 (local address 127.23.44.2:41793)
0929 16:31:22.371358 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.371373 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.371459 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.372076 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.372086 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.372715 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:22.373751 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.373775 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.375062 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.375065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.376754 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.376769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.377952 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.377960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.378252 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.379183 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:22.379205 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:22.381300 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:22.384504 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.384512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.384535 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.384938 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.385325 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.385328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.385330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.385763 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.386333 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:22.386343 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:22.386733 (+ 390us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.387606 (+ 873us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.387786 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":213,"spinlock_wait_cycles":9344,"thread_start_us":74,"threads_started":1}
I20260929 16:31:22.388629 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.369242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.2:58953)
0929 16:31:22.370267 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.370306 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.371361 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.372111 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.372114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.372951 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:22.373747 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.373775 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.374809 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.374813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.376718 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.376733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.377602 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.377606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.377801 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.378645 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:22.378724 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:22.381300 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:22.384949 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.384953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.384957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.385757 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.386168 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.386171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.386173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.386315 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.386787 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:22.386790 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:22.386877 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.388262 (+ 1385us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.388425 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":164,"spinlock_wait_cycles":2432,"thread_start_us":604,"threads_started":1}
I20260929 16:31:22.389720 30940 heartbeater.cc:347] Connected to a master server at 127.23.44.61:43217
I20260929 16:31:22.389780 30939 heartbeater.cc:347] Connected to a master server at 127.23.44.62:44913
I20260929 16:31:22.389853 30938 heartbeater.cc:347] Connected to a master server at 127.23.44.60:38823
I20260929 16:31:22.389957 30939 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:22.389958 30938 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:22.389958 30940 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:22.390597 30938 heartbeater.cc:511] Master 127.23.44.60:38823 requested a full tablet report, sending...
I20260929 16:31:22.390586 30939 heartbeater.cc:511] Master 127.23.44.62:44913 requested a full tablet report, sending...
I20260929 16:31:22.390586 30940 heartbeater.cc:511] Master 127.23.44.61:43217 requested a full tablet report, sending...
I20260929 16:31:22.391781 29838 ts_manager.cc:194] Re-registered known tserver with Master: c9b7e6cec6524c3082a32c7df0f4ab6c (127.23.44.2:38115)
I20260929 16:31:22.391850 30430 ts_manager.cc:194] Registered new tserver with Master: c9b7e6cec6524c3082a32c7df0f4ab6c (127.23.44.2:38115)
I20260929 16:31:22.391904 30622 ts_manager.cc:194] Registered new tserver with Master: c9b7e6cec6524c3082a32c7df0f4ab6c (127.23.44.2:38115)
I20260929 16:31:22.392568 29838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:58953
I20260929 16:31:22.415866 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.411622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:38823 (local address 127.0.0.1:52320)
0929 16:31:22.411796 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.411818 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.411926 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.412297 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.412299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.412318 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:22.412418 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.412424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.413483 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.413485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.414286 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.414295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.415319 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.415324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.415445 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.415486 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.415607 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.415700 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 16:31:22.415879 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.411695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.23.44.60:38823)
0929 16:31:22.411831 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.411837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.411898 (+ 61us) server_negotiation.cc:407] Connection header received
0929 16:31:22.412029 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.412033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.412099 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.412222 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:22.412534 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.413326 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.414411 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.415067 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.415178 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.415329 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.415644 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.415735 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:31:22.416987 30632 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 16:31:22.417675 30661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 939af390c9f94d3c93c53b23f0754cc2. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:22.417886 30661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: This master's current role is: LEARNER
W20260929 16:31:22.509536 30945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:22.509778 30945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:22.509819 30945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:22.513906 30945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:22.513964 30945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:22.514022 30945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:22.519184 30945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab
--principal=oryx/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:45979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=34255
--tserver_master_addrs=127.23.44.62:44913,127.23.44.61:43217,127.23.44.60:38823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:22.520257 30945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:22.521171 30945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:22.530931 30953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:22.530925 30951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:22.531044 30945 server_base.cc:1034] running on GCE node
W20260929 16:31:22.530995 30950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:22.531533 30945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:22.532194 30945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:22.533339 30945 hybrid_clock.cc:648] HybridClock initialized: now 1790699482533308 us; error 46 us; skew 500 ppm
Sep 29 16:31:22 dist-test-slave-mlmz krb5kdc[29793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699482, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:22.536211 30945 init.cc:377] Logged in from keytab as oryx/127.23.44.3@KRBTEST.COM (short username oryx)
I20260929 16:31:22.537039 30945 webserver.cc:460] Webserver started at http://127.23.44.3:34255/ using document root <none> and password file <none>
I20260929 16:31:22.537647 30945 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:22.537724 30945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:22.541452 30945 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 16:31:22.543655 30960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:22.545496 30945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:31:22.545629 30945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "e7511a2044854916bdeb38c7078e48d9"
format_stamp: "Formatted at 2026-09-29 16:31:21 on dist-test-slave-mlmz"
server_key: "41f3224161cd780497964bb4b041ebe3"
server_key_iv: "aa782206ac56af4acdc96d491cab89ae"
server_key_version: "encryptionkey@0"
I20260929 16:31:22.546235 30945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:22.579775 30945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:22.583564 30945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:22.583796 30945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:22.584534 30945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:22.585553 30945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:22.585629 30945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:22.585708 30945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:22.585765 30945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:22.595966 30945 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:45979
I20260929 16:31:22.595984 31073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:45979 every 8 connection(s)
I20260929 16:31:22.597038 30945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:22.606606 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 30945
Sep 29 16:31:22 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699482, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
Sep 29 16:31:22 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699482, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.61@KRBTEST.COM
Sep 29 16:31:22 dist-test-slave-mlmz krb5kdc[29793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699482, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.60@KRBTEST.COM
I20260929 16:31:22.617914 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.600971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:41389 (local address 127.23.44.61:43217)
0929 16:31:22.601254 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.601261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.601285 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:31:22.602404 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.602408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.602480 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.602583 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.604265 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.605190 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.607101 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.607496 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.607596 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.612831 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.612835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.612838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.612866 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.615087 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.616168 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.616175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.616180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.616264 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.616621 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.616625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.616626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.616855 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.616937 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.617601 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.617727 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:31:22.618615 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.601548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:48497 (local address 127.23.44.60:38823)
0929 16:31:22.601684 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.601688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.601703 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:31:22.602779 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.602782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.602843 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.602928 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.604241 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.604959 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.607026 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.607367 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.607444 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.612699 (+ 5255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.612706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.612713 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.612748 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.615199 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.615911 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.615914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.615916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.616011 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.616463 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.616466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.616468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.616713 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.616808 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.618316 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.618443 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:31:22.619182 31080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.601146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:38823 (local address 127.23.44.3:48497)
0929 16:31:22.601634 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.601648 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.602126 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.602990 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.602996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.603452 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:22.604082 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.604091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.605108 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.605114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.606857 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.606863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.607645 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.607649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.608460 (+ 811us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.609213 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:22.609230 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:22.612514 (+ 3284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:22.615316 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.615322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.615326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.615776 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.616129 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.616135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.616141 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.616348 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.616881 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:22.616892 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:22.617289 (+ 397us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.617517 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.617883 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":930,"spinlock_wait_cycles":41600,"thread_start_us":83,"threads_started":1}
I20260929 16:31:22.619182 31079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.600736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:43217 (local address 127.23.44.3:41389)
0929 16:31:22.601085 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.601101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.602127 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.602766 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.602773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.603452 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:22.604081 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.604091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.605400 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.605402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.606846 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.606857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.607808 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.607814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.608161 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.609080 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:22.609107 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:22.612667 (+ 3560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:22.615237 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.615247 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.615269 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.616051 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.616395 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.616400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.616405 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.616496 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.616992 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:22.616995 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:22.617289 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.618075 (+ 786us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.618218 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":386,"spinlock_wait_cycles":47360,"thread_start_us":131,"threads_started":1}
I20260929 16:31:22.619475 31078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.599346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.3:56589)
0929 16:31:22.600505 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.600593 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.602126 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.602766 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.602775 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.603424 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:22.604071 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.604084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.605327 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.605331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.606850 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.606860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.607645 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.607649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.608080 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.609145 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:22.609160 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:22.612744 (+ 3584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:22.615386 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.615389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.615391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.615776 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.616153 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:22.616156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:22.616158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:22.617289 (+ 1131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:22.617822 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:22.617827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:22.617929 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.618025 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.618154 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":252,"spinlock_wait_cycles":21632,"thread_start_us":582,"threads_started":1}
I20260929 16:31:22.620698 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.598899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:56589 (local address 127.23.44.62:44913)
0929 16:31:22.599216 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.599220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.600823 (+ 1603us) server_negotiation.cc:407] Connection header received
0929 16:31:22.602495 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.602498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.602555 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.602656 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.604288 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.605171 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.606992 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.607367 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.607444 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.612857 (+ 5413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.612860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.612863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.612891 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.615269 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.615907 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.615912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.615914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.616036 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.617414 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.617420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.617422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.617665 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.617776 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.620389 (+ 2613us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.620526 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"spinlock_wait_cycles":48000}
I20260929 16:31:22.621320 31076 heartbeater.cc:347] Connected to a master server at 127.23.44.61:43217
I20260929 16:31:22.621578 31076 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:22.622889 31075 heartbeater.cc:347] Connected to a master server at 127.23.44.62:44913
I20260929 16:31:22.623023 31075 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:22.623667 31075 heartbeater.cc:511] Master 127.23.44.62:44913 requested a full tablet report, sending...
I20260929 16:31:22.621232 31074 heartbeater.cc:347] Connected to a master server at 127.23.44.60:38823
I20260929 16:31:22.624310 31074 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:22.624541 31074 heartbeater.cc:511] Master 127.23.44.60:38823 requested a full tablet report, sending...
I20260929 16:31:22.624835 31076 heartbeater.cc:511] Master 127.23.44.61:43217 requested a full tablet report, sending...
I20260929 16:31:22.626086 31085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.608944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.23.44.62:44913)
0929 16:31:22.611346 (+ 2402us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.611351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.611364 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:31:22.611400 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.611403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.611446 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.611541 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.612846 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.613672 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.614974 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.615419 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.615892 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.620929 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.620935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.620941 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.620975 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.622877 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.623935 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.623939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.623941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.624035 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.624497 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.624500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.624502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.624899 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.625697 (+ 798us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.625780 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.625928 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2320,"thread_start_us":69,"threads_started":1}
I20260929 16:31:22.626526 30622 ts_manager.cc:194] Registered new tserver with Master: e7511a2044854916bdeb38c7078e48d9 (127.23.44.3:45979)
I20260929 16:31:22.626086 29838 ts_manager.cc:194] Re-registered known tserver with Master: e7511a2044854916bdeb38c7078e48d9 (127.23.44.3:45979)
I20260929 16:31:22.627552 31089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.610431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.23.44.60:38823)
0929 16:31:22.610657 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.610661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.611068 (+ 407us) server_negotiation.cc:407] Connection header received
0929 16:31:22.611170 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.611173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.611224 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.611303 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.612208 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.613407 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.618716 (+ 5309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.619041 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.619113 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.623189 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.623193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.623196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.623220 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.625340 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.626045 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.626050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.626053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.626190 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.626524 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.626529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.626532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.626806 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.626926 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.627257 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.627389 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
I20260929 16:31:22.627535 30430 ts_manager.cc:194] Registered new tserver with Master: e7511a2044854916bdeb38c7078e48d9 (127.23.44.3:45979)
I20260929 16:31:22.628250 29838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:56589
I20260929 16:31:22.630890 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.609728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.23.44.61:43217)
0929 16:31:22.610032 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.610036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.611098 (+ 1062us) server_negotiation.cc:407] Connection header received
0929 16:31:22.617026 (+ 5928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.617031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.617088 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.617173 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.618005 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.619421 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.621100 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.621536 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.621649 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.624487 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.624492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.624495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.624525 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.628719 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.629531 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.629535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.629537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.629625 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.629920 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.629923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.629925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.630131 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.630262 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.630581 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.630717 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":94,"threads_started":1}
I20260929 16:31:22.630414 29838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:22.634003 29838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:22.636749 30585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: NON_VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: true } } }
I20260929 16:31:22.636924 30585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:22.640754 30585 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2: attempting to promote NON_VOTER 09d573aa99e74de599580ced2ce89990 to VOTER
I20260929 16:31:22.641167 30585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } }
I20260929 16:31:22.644274 30447 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 FOLLOWER]: Refusing update from remote peer 939af390c9f94d3c93c53b23f0754cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 16:31:22.644266 30632 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 1 LEARNER]: Refusing update from remote peer 939af390c9f94d3c93c53b23f0754cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 16:31:22.644719 31092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 16:31:22.644881 31093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 16:31:22.647163 30585 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 09d573aa99e74de599580ced2ce89990 (127.23.44.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } } }
I20260929 16:31:22.647817 30447 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 09d573aa99e74de599580ced2ce89990 (127.23.44.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } } }
I20260929 16:31:22.647801 30632 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 09d573aa99e74de599580ced2ce89990 (127.23.44.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } } }
I20260929 16:31:22.648777 30585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } } }
I20260929 16:31:22.648897 30585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:22.649029 30585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } } }
I20260929 16:31:22.649104 30585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:22.649216 30585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } } }
I20260929 16:31:22.649329 30585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:22.649600 31101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } } }
I20260929 16:31:22.649746 31101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:31:22.650175 30661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "939af390c9f94d3c93c53b23f0754cc2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } } }
I20260929 16:31:22.650312 30661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:31:22.656975 31102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.651944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.23.44.2:38115)
0929 16:31:22.652238 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.652253 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.652275 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:31:22.652353 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.652359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.652600 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.653060 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:22.653183 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.654227 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.655504 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.656140 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.656249 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.656403 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.656727 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.656825 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":116,"threads_started":1}
I20260929 16:31:22.656960 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.651726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:38115 (local address 127.0.0.1:56420)
0929 16:31:22.651891 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.651907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.651988 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.652767 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.652770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.652801 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:22.652903 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.652910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.654333 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.654338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.655397 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.655404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.656462 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.656465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.656562 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.656590 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.656693 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.656825 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":41}
I20260929 16:31:22.658345 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.652090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:36317 (local address 127.0.0.1:59126)
0929 16:31:22.652393 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.652409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.652907 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.653813 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.653816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.653833 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:22.653934 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.653939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.654935 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.654942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.656353 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.656364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.657896 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.657900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.658052 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.658077 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.658147 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.658211 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":50,"thread_start_us":100,"threads_started":1}
I20260929 16:31:22.658612 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.652657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.23.44.1:36317)
0929 16:31:22.653395 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.653413 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.653440 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:31:22.653519 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.653523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.653676 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.653798 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:22.654031 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.654790 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.656506 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.657738 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.657952 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.658125 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.658180 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.658272 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":117,"threads_started":1}
I20260929 16:31:22.659533 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.652961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:45979 (local address 127.0.0.1:36604)
0929 16:31:22.653347 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.653360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.653462 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.655053 (+ 1591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.655056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.655071 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:22.655173 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.655179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.656621 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.656624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.657935 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.657943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.659102 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.659106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.659189 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.659211 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.659301 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.659391 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260929 16:31:22.659587 31104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.652546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.23.44.3:45979)
0929 16:31:22.654502 (+ 1956us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.654513 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.654534 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:31:22.654603 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.654608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.654783 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.654932 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:22.655303 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.656487 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.658068 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.658983 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.659077 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.659172 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.659314 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.659421 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1860,"thread_start_us":105,"threads_started":1}
I20260929 16:31:22.660020 30872 tablet_service.cc:1467] Processing CreateTablet for tablet c693e2af96694bbdb9db6fa5bb10544c (DEFAULT_TABLE table=test-table [id=926395012f95472592f1e991748ca4aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:22.661438 30872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c693e2af96694bbdb9db6fa5bb10544c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:22.662833 30735 tablet_service.cc:1467] Processing CreateTablet for tablet c693e2af96694bbdb9db6fa5bb10544c (DEFAULT_TABLE table=test-table [id=926395012f95472592f1e991748ca4aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:22.663077 31008 tablet_service.cc:1467] Processing CreateTablet for tablet c693e2af96694bbdb9db6fa5bb10544c (DEFAULT_TABLE table=test-table [id=926395012f95472592f1e991748ca4aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:22.663748 30735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c693e2af96694bbdb9db6fa5bb10544c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:22.664002 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c693e2af96694bbdb9db6fa5bb10544c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:22.670742 31108 tablet_bootstrap.cc:492] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52: Bootstrap starting.
I20260929 16:31:22.670778 31107 tablet_bootstrap.cc:492] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c: Bootstrap starting.
I20260929 16:31:22.671902 31109 tablet_bootstrap.cc:492] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9: Bootstrap starting.
I20260929 16:31:22.672997 31108 tablet_bootstrap.cc:654] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:22.672997 31107 tablet_bootstrap.cc:654] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:22.673902 31107 log.cc:824] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:22.673902 31108 log.cc:824] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:22.673949 31109 tablet_bootstrap.cc:654] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:22.674757 31109 log.cc:824] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:22.675655 31108 tablet_bootstrap.cc:492] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52: No bootstrap required, opened a new log
I20260929 16:31:22.675853 31108 ts_tablet_manager.cc:1397] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260929 16:31:22.676481 31109 tablet_bootstrap.cc:492] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9: No bootstrap required, opened a new log
I20260929 16:31:22.676677 31109 ts_tablet_manager.cc:1397] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:31:22.679095 31108 raft_consensus.cc:348] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } } peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } }
I20260929 16:31:22.679346 31108 raft_consensus.cc:374] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:22.679423 31108 raft_consensus.cc:729] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30dfcbab3ef543848c69b4734509af52, State: Initialized, Role: FOLLOWER
I20260929 16:31:22.679921 31108 consensus_queue.cc:260] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } } peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } }
I20260929 16:31:22.680903 31108 ts_tablet_manager.cc:1428] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 16:31:22.681402 31109 raft_consensus.cc:348] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } } peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } }
I20260929 16:31:22.681636 31109 raft_consensus.cc:374] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:22.681689 31109 raft_consensus.cc:729] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7511a2044854916bdeb38c7078e48d9, State: Initialized, Role: FOLLOWER
I20260929 16:31:22.681694 30802 heartbeater.cc:503] Master 127.23.44.62:44913 was elected leader, sending a full tablet report...
I20260929 16:31:22.682195 31109 consensus_queue.cc:260] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } } peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } }
I20260929 16:31:22.683310 31109 ts_tablet_manager.cc:1428] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 16:31:22.683328 31107 tablet_bootstrap.cc:492] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c: No bootstrap required, opened a new log
I20260929 16:31:22.683615 31107 ts_tablet_manager.cc:1397] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20260929 16:31:22.683804 31075 heartbeater.cc:503] Master 127.23.44.62:44913 was elected leader, sending a full tablet report...
I20260929 16:31:22.688488 31107 raft_consensus.cc:348] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } } peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } }
I20260929 16:31:22.688695 31107 raft_consensus.cc:374] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:22.688730 31107 raft_consensus.cc:729] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9b7e6cec6524c3082a32c7df0f4ab6c, State: Initialized, Role: FOLLOWER
I20260929 16:31:22.689142 31107 consensus_queue.cc:260] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } } peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } }
I20260929 16:31:22.690183 31107 ts_tablet_manager.cc:1428] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:31:22.690567 30939 heartbeater.cc:503] Master 127.23.44.62:44913 was elected leader, sending a full tablet report...
I20260929 16:31:22.837841 31115 raft_consensus.cc:482] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:22.838013 31115 raft_consensus.cc:504] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } } peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } }
I20260929 16:31:22.839082 31115 leader_election.cc:288] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30dfcbab3ef543848c69b4734509af52 (127.23.44.1:36317), e7511a2044854916bdeb38c7078e48d9 (127.23.44.3:45979)
I20260929 16:31:22.843158 30943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.839255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:36317 (local address 127.23.44.2:37959)
0929 16:31:22.839415 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.839431 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.839546 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.839845 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.839849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.839870 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:22.839998 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.840008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.840826 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.840829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.841948 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.841958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.842671 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.842676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.842804 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.842845 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.842931 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.843008 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":47}
I20260929 16:31:22.843400 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.839444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:37959 (local address 127.23.44.1:36317)
0929 16:31:22.839580 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.839585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.839600 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:31:22.839653 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.839656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.839716 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.839790 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:22.840122 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.840703 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.842085 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.842547 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.842627 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.842675 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.843089 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.843222 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:31:22.843570 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.839526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:45979 (local address 127.23.44.2:56839)
0929 16:31:22.839656 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:22.839668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:22.839732 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:22.840110 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:22.840113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:22.840131 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:22.840225 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.840231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.841172 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.841174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.842086 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:22.842096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.843097 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.843101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:22.843218 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.843242 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:31:22.843344 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:31:22.843438 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":2}
I20260929 16:31:22.843752 31104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.839704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:56839 (local address 127.23.44.3:45979)
0929 16:31:22.839849 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.839854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.839868 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:31:22.839924 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.839927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.839980 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.840074 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:22.840347 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.841047 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.842225 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.842769 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.842944 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.843005 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.843503 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.843598 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:31:22.844079 30755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c693e2af96694bbdb9db6fa5bb10544c" candidate_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30dfcbab3ef543848c69b4734509af52" is_pre_election: true
I20260929 16:31:22.844276 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c693e2af96694bbdb9db6fa5bb10544c" candidate_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7511a2044854916bdeb38c7078e48d9" is_pre_election: true
I20260929 16:31:22.844388 30755 raft_consensus.cc:2454] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9b7e6cec6524c3082a32c7df0f4ab6c in term 0.
I20260929 16:31:22.844571 31028 raft_consensus.cc:2454] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9b7e6cec6524c3082a32c7df0f4ab6c in term 0.
I20260929 16:31:22.844936 30828 leader_election.cc:302] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30dfcbab3ef543848c69b4734509af52, c9b7e6cec6524c3082a32c7df0f4ab6c; no voters:
I20260929 16:31:22.845238 31115 raft_consensus.cc:2781] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:22.845336 31115 raft_consensus.cc:482] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:22.845414 31115 raft_consensus.cc:3037] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:22.846899 31115 raft_consensus.cc:504] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } } peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } }
I20260929 16:31:22.847265 31115 leader_election.cc:288] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [CANDIDATE]: Term 1 election: Requested vote from peers 30dfcbab3ef543848c69b4734509af52 (127.23.44.1:36317), e7511a2044854916bdeb38c7078e48d9 (127.23.44.3:45979)
I20260929 16:31:22.847738 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c693e2af96694bbdb9db6fa5bb10544c" candidate_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7511a2044854916bdeb38c7078e48d9"
I20260929 16:31:22.847949 31028 raft_consensus.cc:3037] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Advancing to term 1
W20260929 16:31:22.848865 31077 tablet.cc:2369] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:31:22.849064 31028 raft_consensus.cc:2454] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9b7e6cec6524c3082a32c7df0f4ab6c in term 1.
I20260929 16:31:22.849179 30755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c693e2af96694bbdb9db6fa5bb10544c" candidate_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30dfcbab3ef543848c69b4734509af52"
I20260929 16:31:22.849349 30755 raft_consensus.cc:3037] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:22.849506 30828 leader_election.cc:302] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9b7e6cec6524c3082a32c7df0f4ab6c, e7511a2044854916bdeb38c7078e48d9; no voters:
I20260929 16:31:22.849697 31115 raft_consensus.cc:2781] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:22.849892 31115 raft_consensus.cc:686] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 1 LEADER]: Becoming Leader. State: Replica: c9b7e6cec6524c3082a32c7df0f4ab6c, State: Running, Role: LEADER
I20260929 16:31:22.850235 31115 consensus_queue.cc:237] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } } peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } }
I20260929 16:31:22.850745 30755 raft_consensus.cc:2454] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9b7e6cec6524c3082a32c7df0f4ab6c in term 1.
I20260929 16:31:22.854029 29838 catalog_manager.cc:5495] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c reported cstate change: term changed from 0 to 1, leader changed from <none> to c9b7e6cec6524c3082a32c7df0f4ab6c (127.23.44.2). New cstate: current_term: 1 leader_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
W20260929 16:31:22.868952 30941 tablet.cc:2369] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:31:22.892181 31102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.888545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.23.44.2:38115)
0929 16:31:22.888707 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.888711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.888726 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:31:22.888880 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.888882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.888945 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.889047 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:22.889396 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.889906 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.890752 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.890979 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.891059 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.891317 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:22.891718 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:22.891810 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.891969 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.892019 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:31:22.897948 31028 raft_consensus.cc:1261] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9 [term 1 FOLLOWER]: Refusing update from remote peer c9b7e6cec6524c3082a32c7df0f4ab6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:22.898182 30755 raft_consensus.cc:1261] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52 [term 1 FOLLOWER]: Refusing update from remote peer c9b7e6cec6524c3082a32c7df0f4ab6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:22.898722 31116 consensus_queue.cc:1035] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:22.898917 31115 consensus_queue.cc:1035] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:31:22.899753 30804 tablet.cc:2369] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:31:22.908946 31121 mvcc.cc:205] Tried to move back new op lower bound from 7334705081942827008 to 7334705081757724672. Current Snapshot: MvccSnapshot[applied={T|T < 7334705081942827008}]
I20260929 16:31:22.908990 31120 mvcc.cc:205] Tried to move back new op lower bound from 7334705081942827008 to 7334705081757724672. Current Snapshot: MvccSnapshot[applied={T|T < 7334705081942827008}]
I20260929 16:31:22.921561 29838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:22.921744 29838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:22.926510 29838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 939af390c9f94d3c93c53b23f0754cc2: Sending DeleteTablet for 3 replicas of tablet c693e2af96694bbdb9db6fa5bb10544c
I20260929 16:31:22.927187 30872 tablet_service.cc:1514] Processing DeleteTablet for tablet c693e2af96694bbdb9db6fa5bb10544c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:22 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:56420
I20260929 16:31:22.927556 31008 tablet_service.cc:1514] Processing DeleteTablet for tablet c693e2af96694bbdb9db6fa5bb10544c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:22 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:36604
I20260929 16:31:22.927708 30735 tablet_service.cc:1514] Processing DeleteTablet for tablet c693e2af96694bbdb9db6fa5bb10544c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:22 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:59126
I20260929 16:31:22.928020 31130 tablet_replica.cc:330] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9: stopping tablet replica
I20260929 16:31:22.928158 31131 tablet_replica.cc:330] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52: stopping tablet replica
I20260929 16:31:22.928351 31130 raft_consensus.cc:2229] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:22.928589 31129 tablet_replica.cc:330] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c: stopping tablet replica
I20260929 16:31:22.928696 31130 raft_consensus.cc:2258] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:22.928979 31129 raft_consensus.cc:2229] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:22.929422 31129 raft_consensus.cc:2258] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:22.930560 31130 ts_tablet_manager.cc:1905] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:22.930640 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 29806
I20260929 16:31:22.930706 31131 raft_consensus.cc:2229] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:22.931074 31131 raft_consensus.cc:2258] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:22.933079 31131 ts_tablet_manager.cc:1905] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:22.934244 31130 ts_tablet_manager.cc:1918] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:22.934407 31130 log.cc:1196] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/c693e2af96694bbdb9db6fa5bb10544c
I20260929 16:31:22.934877 31130 ts_tablet_manager.cc:1939] T c693e2af96694bbdb9db6fa5bb10544c P e7511a2044854916bdeb38c7078e48d9: Deleting consensus metadata
W20260929 16:31:22.938897 30963 connection.cc:535] server connection from 127.0.0.1:36604 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:31:22.941032 31129 ts_tablet_manager.cc:1905] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:22.942469 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.929330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39004 (local address 127.23.44.61:43217)
0929 16:31:22.929441 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.929448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.931517 (+ 2069us) server_negotiation.cc:407] Connection header received
0929 16:31:22.931709 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.931715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.931790 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.931846 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.932985 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.933921 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.935330 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.935790 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.935885 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.938585 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.938591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.938598 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.938632 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.940482 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.940993 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.940996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.940998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.941090 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.941445 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.941450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.941454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.941679 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.941796 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.942151 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.942293 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 16:31:22.944020 31129 ts_tablet_manager.cc:1918] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:22.944165 31129 log.cc:1196] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/c693e2af96694bbdb9db6fa5bb10544c
I20260929 16:31:22.945031 31129 ts_tablet_manager.cc:1939] T c693e2af96694bbdb9db6fa5bb10544c P c9b7e6cec6524c3082a32c7df0f4ab6c: Deleting consensus metadata
I20260929 16:31:22.945757 31131 ts_tablet_manager.cc:1918] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:22.945890 31131 log.cc:1196] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-0/wal/wals/c693e2af96694bbdb9db6fa5bb10544c
I20260929 16:31:22.946291 31131 ts_tablet_manager.cc:1939] T c693e2af96694bbdb9db6fa5bb10544c P 30dfcbab3ef543848c69b4734509af52: Deleting consensus metadata
W20260929 16:31:22.946475 30827 connection.cc:441] server connection from 127.0.0.1:56420 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56420 (request call id 1) could send its response
W20260929 16:31:22.947472 30689 connection.cc:441] server connection from 127.0.0.1:59126 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59126 (request call id 1) could send its response
I20260929 16:31:22.955060 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.943533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.23.44.61:43217)
0929 16:31:22.943923 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.943928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.944896 (+ 968us) server_negotiation.cc:407] Connection header received
0929 16:31:22.945080 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.945083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.945128 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.945179 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.946661 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.947452 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.948858 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.949222 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.949317 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.951205 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.951209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.951212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.951240 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.953088 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.953628 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.953632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.953634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.953720 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.954064 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.954067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.954069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.954284 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.954388 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.954750 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.954885 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299}
I20260929 16:31:22.957382 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.944123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.23.44.60:38823)
0929 16:31:22.944343 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.944350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.946936 (+ 2586us) server_negotiation.cc:407] Connection header received
0929 16:31:22.947093 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.947097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.947138 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.947196 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.948412 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.949190 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.950593 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.951020 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.951107 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.953208 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.953216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.953221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.953252 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.955177 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.955686 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.955690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.955692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.955761 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.956120 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.956124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.956127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.956346 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.956492 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.957068 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.957191 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:31:22.957613 31089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:22.930177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52328 (local address 127.23.44.60:38823)
0929 16:31:22.930307 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:22.930311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:22.944850 (+ 14539us) server_negotiation.cc:407] Connection header received
0929 16:31:22.945110 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:22.945114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:22.945190 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:22.945288 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:22.946284 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.947146 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.948433 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:22.948803 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:22.948881 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:22.951265 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:22.951269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:22.951272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:22.951295 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:22.954511 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.955103 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.955108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.955110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.955195 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:22.955566 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:22.955570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:22.955572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:22.955818 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:22.955934 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:22.957229 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0929 16:31:22.957441 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 16:31:23.913450 31141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:23.912862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.1:50851)
0929 16:31:23.913192 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:23.913326 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":76,"threads_started":1}
I20260929 16:31:23.913769 31140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:23.912862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.3:57917)
0929 16:31:23.913589 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:23.913672 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":77,"threads_started":1}
W20260929 16:31:23.914032 30802 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:44913 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
W20260929 16:31:23.914289 31075 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:44913 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
I20260929 16:31:23.915333 31142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:23.914933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.2:41713)
0929 16:31:23.915163 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:23.915251 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
W20260929 16:31:23.915540 30939 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:44913 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
I20260929 16:31:24.426098 31143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 939af390c9f94d3c93c53b23f0754cc2)
I20260929 16:31:24.426359 31143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } }
I20260929 16:31:24.427013 31144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 939af390c9f94d3c93c53b23f0754cc2)
I20260929 16:31:24.427166 31144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } }
I20260929 16:31:24.427448 31143 leader_election.cc:288] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 939af390c9f94d3c93c53b23f0754cc2 (127.23.44.62:44913), 09d573aa99e74de599580ced2ce89990 (127.23.44.60:38823)
I20260929 16:31:24.428042 31144 leader_election.cc:288] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 939af390c9f94d3c93c53b23f0754cc2 (127.23.44.62:44913), 7ad0e747da51470fbc4a8d4458498b64 (127.23.44.61:43217)
I20260929 16:31:24.428191 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.427526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:53078)
0929 16:31:24.427998 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.428081 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":55,"threads_started":1}
W20260929 16:31:24.428360 30424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
I20260929 16:31:24.429102 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.428320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:53088)
0929 16:31:24.428828 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.428908 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":73,"threads_started":1}
W20260929 16:31:24.429319 30608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
I20260929 16:31:24.429454 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.428956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:53094)
0929 16:31:24.429332 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.429385 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
W20260929 16:31:24.429580 30424 leader_election.cc:334] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 939af390c9f94d3c93c53b23f0754cc2 (127.23.44.62:44913): Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
I20260929 16:31:24.430224 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.429990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:53108)
0929 16:31:24.430085 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.430147 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 16:31:24.430361 30608 leader_election.cc:334] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 939af390c9f94d3c93c53b23f0754cc2 (127.23.44.62:44913): Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
I20260929 16:31:24.432255 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.427806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:38823 (local address 127.0.0.1:52344)
0929 16:31:24.428019 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.428031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:24.428120 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:24.428558 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:24.428562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:24.428588 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:24.428718 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.428729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.430188 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.430191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.431103 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.431112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.431843 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.431846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.431937 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.431967 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:31:24.432054 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:31:24.432128 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260929 16:31:24.432299 31147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.428038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.23.44.60:38823)
0929 16:31:24.428337 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:24.428341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:24.428355 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:31:24.428400 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:24.428403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:24.428450 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:24.428538 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:24.428858 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.429995 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.431270 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.431721 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.431842 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.431900 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:24.432082 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 16:31:24.432152 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":231,"thread_start_us":76,"threads_started":1}
I20260929 16:31:24.432680 30632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ad0e747da51470fbc4a8d4458498b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "09d573aa99e74de599580ced2ce89990" is_pre_election: true
I20260929 16:31:24.432899 30632 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ad0e747da51470fbc4a8d4458498b64 in term 1.
I20260929 16:31:24.432981 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.429087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.23.44.61:43217)
0929 16:31:24.429412 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:24.429421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:24.429433 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:31:24.429762 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:24.429765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:24.429810 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:24.429900 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:24.430143 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.430829 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.431890 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.432336 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.432451 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.432504 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:24.432790 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:31:24.432858 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
I20260929 16:31:24.433043 31150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.428670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:43217 (local address 127.0.0.1:39030)
0929 16:31:24.429038 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.429053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:24.429156 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:24.429931 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:24.429934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:24.429949 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:24.430041 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.430048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.430948 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.430951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.431764 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.431772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.432534 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.432537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.432637 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.432656 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 16:31:24.432761 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 16:31:24.432841 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260929 16:31:24.433238 30424 leader_election.cc:302] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09d573aa99e74de599580ced2ce89990, 7ad0e747da51470fbc4a8d4458498b64; no voters: 939af390c9f94d3c93c53b23f0754cc2
I20260929 16:31:24.433466 31143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 16:31:24.433542 31143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 939af390c9f94d3c93c53b23f0754cc2)
I20260929 16:31:24.433596 31143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:31:24.433756 30447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09d573aa99e74de599580ced2ce89990" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7ad0e747da51470fbc4a8d4458498b64" is_pre_election: true
I20260929 16:31:24.434656 31143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } }
I20260929 16:31:24.434906 30447 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09d573aa99e74de599580ced2ce89990 in current term 2: Already voted for candidate 7ad0e747da51470fbc4a8d4458498b64 in this term.
I20260929 16:31:24.435083 31143 leader_election.cc:288] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [CANDIDATE]: Term 2 election: Requested vote from peers 939af390c9f94d3c93c53b23f0754cc2 (127.23.44.62:44913), 09d573aa99e74de599580ced2ce89990 (127.23.44.60:38823)
I20260929 16:31:24.435601 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.435245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:53114)
0929 16:31:24.435379 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.435438 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 16:31:24.435602 30632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ad0e747da51470fbc4a8d4458498b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "09d573aa99e74de599580ced2ce89990"
I20260929 16:31:24.435760 30632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:31:24.436650 30632 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ad0e747da51470fbc4a8d4458498b64 in term 2.
I20260929 16:31:24.437006 30424 leader_election.cc:302] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d573aa99e74de599580ced2ce89990, 7ad0e747da51470fbc4a8d4458498b64; no voters:
I20260929 16:31:24.437062 30608 leader_election.cc:302] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09d573aa99e74de599580ced2ce89990; no voters: 7ad0e747da51470fbc4a8d4458498b64, 939af390c9f94d3c93c53b23f0754cc2
I20260929 16:31:24.437336 31144 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:31:24.437465 31143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:31:24.437585 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.437348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:53118)
0929 16:31:24.437461 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.437519 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 16:31:24.437668 31143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [term 2 LEADER]: Becoming Leader. State: Replica: 7ad0e747da51470fbc4a8d4458498b64, State: Running, Role: LEADER
I20260929 16:31:24.437929 31143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } }
W20260929 16:31:24.438836 30424 leader_election.cc:334] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 939af390c9f94d3c93c53b23f0754cc2 (127.23.44.62:44913): Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
I20260929 16:31:24.438835 31153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad0e747da51470fbc4a8d4458498b64. Latest consensus state: current_term: 2 leader_uuid: "7ad0e747da51470fbc4a8d4458498b64" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } } }
I20260929 16:31:24.439007 31153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:24.439329 31155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:24.441510 31155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:24.441900 31155 catalog_manager.cc:1259] Loaded cluster ID: e33f563b1bbe46d5a0c2b73bb61906b7
I20260929 16:31:24.441941 31155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:24.442482 31155 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:24.442909 31155 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: Loaded TSK: 0
I20260929 16:31:24.443552 31155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:31:24.485194 30432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:24.488240 30432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:24.491350 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.491072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:53140)
0929 16:31:24.491211 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.491282 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:31:24.491648 30632 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [term 2 FOLLOWER]: Refusing update from remote peer 7ad0e747da51470fbc4a8d4458498b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 16:31:24.492136 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.491932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.0.0.1:53144)
0929 16:31:24.492035 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.492082 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 16:31:24.492513 31153 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 16:31:24.492817 30424 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 -> Peer 939af390c9f94d3c93c53b23f0754cc2 (127.23.44.62:44913): Couldn't send request to peer 939af390c9f94d3c93c53b23f0754cc2. Status: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:31:24.494633 31144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad0e747da51470fbc4a8d4458498b64. Latest consensus state: current_term: 2 leader_uuid: "7ad0e747da51470fbc4a8d4458498b64" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } } }
I20260929 16:31:24.494786 31144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:31:24.495796 31156 mvcc.cc:205] Tried to move back new op lower bound from 7334705088471121920 to 7334705088259981312. Current Snapshot: MvccSnapshot[applied={T|T < 7334705088471121920}]
I20260929 16:31:24.496019 31143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7ad0e747da51470fbc4a8d4458498b64" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } } }
I20260929 16:31:24.496163 31143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:24.496611 31144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7ad0e747da51470fbc4a8d4458498b64" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "939af390c9f94d3c93c53b23f0754cc2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44913 } } peers { permanent_uuid: "7ad0e747da51470fbc4a8d4458498b64" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 43217 } attrs { promote: false } } peers { permanent_uuid: "09d573aa99e74de599580ced2ce89990" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 38823 } attrs { promote: false } } }
I20260929 16:31:24.496717 31144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d573aa99e74de599580ced2ce89990 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:31:24.507843 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.502759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:36317 (local address 127.0.0.1:59136)
0929 16:31:24.503546 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.503563 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:24.503686 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:24.503985 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:24.503988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:24.504003 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:24.504159 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.504168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.505293 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.505299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.506498 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.506508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.507382 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.507386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.507484 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.507510 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:31:24.507591 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:31:24.507671 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":23}
I20260929 16:31:24.508263 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.502876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.23.44.1:36317)
0929 16:31:24.503448 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:24.503453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:24.503692 (+ 239us) server_negotiation.cc:407] Connection header received
0929 16:31:24.503794 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:24.503797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:24.503851 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:24.504291 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:24.504405 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.505148 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.506635 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.507264 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.507364 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.507432 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:24.508031 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 16:31:24.508116 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":89,"threads_started":1}
I20260929 16:31:24.508266 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.503232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:38115 (local address 127.0.0.1:56432)
0929 16:31:24.503837 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.503852 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:24.503923 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:24.504368 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:24.504371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:24.504391 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:24.504477 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.504483 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.505620 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.505624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.506974 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.506982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.507869 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.507872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.507962 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.507987 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:31:24.508051 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:31:24.508118 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":174,"thread_start_us":121,"threads_started":1}
I20260929 16:31:24.508786 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.503001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:45979 (local address 127.0.0.1:36612)
0929 16:31:24.503599 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.503614 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:24.503693 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:24.504210 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:24.504213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:24.504229 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:24.504342 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.504351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.505542 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.505549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.507434 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.507445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.508397 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.508400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.508495 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.508517 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:31:24.508591 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:31:24.508676 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":183}
I20260929 16:31:24.508874 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.503147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.23.44.3:45979)
0929 16:31:24.503441 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:24.503447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:24.503787 (+ 340us) server_negotiation.cc:407] Connection header received
0929 16:31:24.503939 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:24.503945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:24.504009 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:24.504089 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:24.504472 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.505408 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.507534 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.508180 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.508274 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.508343 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:24.508640 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 16:31:24.508724 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260929 16:31:24.508877 30735 tablet_service.cc:1467] Processing CreateTablet for tablet 18432db9494943328ca1001d0ec426d7 (DEFAULT_TABLE table=test-table [id=3e79126406ab46928ac10d5de3599e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:24.509163 30735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18432db9494943328ca1001d0ec426d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:24.510046 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.503340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.23.44.2:38115)
0929 16:31:24.504132 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:24.504137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:24.504151 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:31:24.504204 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:24.504207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:24.504253 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:24.504360 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:24.504587 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.505510 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.507071 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.507751 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.509678 (+ 1927us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.509731 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:24.509761 (+ 30us) server_negotiation.cc:299] Negotiation successful
0929 16:31:24.509848 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":85,"threads_started":1}
I20260929 16:31:24.510025 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 18432db9494943328ca1001d0ec426d7 (DEFAULT_TABLE table=test-table [id=3e79126406ab46928ac10d5de3599e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:24.510303 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18432db9494943328ca1001d0ec426d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:24.510486 30872 tablet_service.cc:1467] Processing CreateTablet for tablet 18432db9494943328ca1001d0ec426d7 (DEFAULT_TABLE table=test-table [id=3e79126406ab46928ac10d5de3599e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:24.510764 30872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18432db9494943328ca1001d0ec426d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:24.511721 31165 tablet_bootstrap.cc:492] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52: Bootstrap starting.
I20260929 16:31:24.512815 31165 tablet_bootstrap.cc:654] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:24.513007 31166 tablet_bootstrap.cc:492] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9: Bootstrap starting.
I20260929 16:31:24.513610 31167 tablet_bootstrap.cc:492] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c: Bootstrap starting.
I20260929 16:31:24.513899 31166 tablet_bootstrap.cc:654] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:24.514463 31165 tablet_bootstrap.cc:492] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52: No bootstrap required, opened a new log
I20260929 16:31:24.514575 31165 ts_tablet_manager.cc:1397] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:31:24.514606 31167 tablet_bootstrap.cc:654] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:24.515025 31166 tablet_bootstrap.cc:492] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9: No bootstrap required, opened a new log
I20260929 16:31:24.515137 31166 ts_tablet_manager.cc:1397] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:31:24.515148 31165 raft_consensus.cc:348] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } } peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } }
I20260929 16:31:24.515293 31165 raft_consensus.cc:374] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:24.515344 31165 raft_consensus.cc:729] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30dfcbab3ef543848c69b4734509af52, State: Initialized, Role: FOLLOWER
I20260929 16:31:24.515501 31165 consensus_queue.cc:260] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } } peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } }
I20260929 16:31:24.515640 31166 raft_consensus.cc:348] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } } peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } }
I20260929 16:31:24.515753 31166 raft_consensus.cc:374] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:24.515795 31166 raft_consensus.cc:729] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7511a2044854916bdeb38c7078e48d9, State: Initialized, Role: FOLLOWER
I20260929 16:31:24.515818 31167 tablet_bootstrap.cc:492] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c: No bootstrap required, opened a new log
I20260929 16:31:24.515893 31165 ts_tablet_manager.cc:1428] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 16:31:24.515906 31167 ts_tablet_manager.cc:1397] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:24.515933 31166 consensus_queue.cc:260] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } } peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } }
I20260929 16:31:24.516359 31167 raft_consensus.cc:348] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } } peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } }
I20260929 16:31:24.516477 31167 raft_consensus.cc:374] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:24.516515 31167 raft_consensus.cc:729] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9b7e6cec6524c3082a32c7df0f4ab6c, State: Initialized, Role: FOLLOWER
I20260929 16:31:24.516649 31167 consensus_queue.cc:260] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } } peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } }
I20260929 16:31:24.517012 31167 ts_tablet_manager.cc:1428] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:24.517319 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.516851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.3:52553)
0929 16:31:24.517135 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.517196 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":96,"threads_started":1}
I20260929 16:31:24.518508 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.518084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.2:51659)
0929 16:31:24.518335 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.518391 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
I20260929 16:31:24.519608 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.519351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.2:35791)
0929 16:31:24.519492 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.519544 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 16:31:24.519997 30803 heartbeater.cc:503] Master 127.23.44.61:43217 was elected leader, sending a full tablet report...
I20260929 16:31:24.520596 30940 heartbeater.cc:503] Master 127.23.44.61:43217 was elected leader, sending a full tablet report...
I20260929 16:31:24.522066 31166 ts_tablet_manager.cc:1428] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260929 16:31:24.522503 31076 heartbeater.cc:503] Master 127.23.44.61:43217 was elected leader, sending a full tablet report...
I20260929 16:31:24.522961 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.522733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.3:41491)
0929 16:31:24.522839 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.522893 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 16:31:24.523042 31175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.517009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.1:45425)
0929 16:31:24.522900 (+ 5891us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.522947 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5826,"thread_start_us":95,"threads_started":1}
I20260929 16:31:24.523826 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.523450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.1:34695)
0929 16:31:24.523690 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.523739 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260929 16:31:24.599367 31172 raft_consensus.cc:482] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:24.599608 31172 raft_consensus.cc:504] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } } peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } }
I20260929 16:31:24.600744 31172 leader_election.cc:288] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30dfcbab3ef543848c69b4734509af52 (127.23.44.1:36317), c9b7e6cec6524c3082a32c7df0f4ab6c (127.23.44.2:38115)
I20260929 16:31:24.604995 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.601179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:38115 (local address 127.23.44.3:49355)
0929 16:31:24.601539 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.601551 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:24.601625 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:24.601997 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:24.602000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:24.602014 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:24.602101 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.602106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.603018 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.603023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.603834 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.603842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.604556 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.604559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.604661 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.604705 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 16:31:24.604782 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 16:31:24.604853 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":179,"threads_started":1}
I20260929 16:31:24.605019 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.601593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:49355 (local address 127.23.44.2:38115)
0929 16:31:24.601707 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:24.601711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:24.601724 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:31:24.601775 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:24.601778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:24.601821 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:24.601895 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:24.602218 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.602913 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.603992 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.604404 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.604545 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.604606 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:24.604799 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 16:31:24.604889 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:31:24.605130 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.600982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:36317 (local address 127.23.44.3:41815)
0929 16:31:24.601142 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.601159 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:24.601301 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:24.601665 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:24.601671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:24.601688 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:24.601796 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.601801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.602748 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.602751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.603834 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:24.603843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.604743 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.604750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:24.604853 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.604875 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:31:24.604944 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:31:24.605008 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":1}
I20260929 16:31:24.605516 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.601104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:41815 (local address 127.23.44.1:36317)
0929 16:31:24.601238 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:24.601243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:24.601278 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:31:24.601398 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:24.601401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:24.601456 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:24.601527 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:24.601910 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.602639 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.603983 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.604528 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.604633 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.604711 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:24.605217 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 16:31:24.605392 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:31:24.605873 30755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18432db9494943328ca1001d0ec426d7" candidate_uuid: "e7511a2044854916bdeb38c7078e48d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30dfcbab3ef543848c69b4734509af52" is_pre_election: true
I20260929 16:31:24.606027 30755 raft_consensus.cc:2454] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7511a2044854916bdeb38c7078e48d9 in term 0.
I20260929 16:31:24.605958 30892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18432db9494943328ca1001d0ec426d7" candidate_uuid: "e7511a2044854916bdeb38c7078e48d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" is_pre_election: true
I20260929 16:31:24.606194 30892 raft_consensus.cc:2454] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7511a2044854916bdeb38c7078e48d9 in term 0.
I20260929 16:31:24.606457 30964 leader_election.cc:302] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30dfcbab3ef543848c69b4734509af52, e7511a2044854916bdeb38c7078e48d9; no voters:
I20260929 16:31:24.606725 31172 raft_consensus.cc:2781] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:24.606778 31172 raft_consensus.cc:482] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:24.606805 31172 raft_consensus.cc:3037] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:24.607914 31172 raft_consensus.cc:504] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } } peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } }
I20260929 16:31:24.608239 31172 leader_election.cc:288] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [CANDIDATE]: Term 1 election: Requested vote from peers 30dfcbab3ef543848c69b4734509af52 (127.23.44.1:36317), c9b7e6cec6524c3082a32c7df0f4ab6c (127.23.44.2:38115)
I20260929 16:31:24.608623 30755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18432db9494943328ca1001d0ec426d7" candidate_uuid: "e7511a2044854916bdeb38c7078e48d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30dfcbab3ef543848c69b4734509af52"
I20260929 16:31:24.608705 30892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18432db9494943328ca1001d0ec426d7" candidate_uuid: "e7511a2044854916bdeb38c7078e48d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c"
I20260929 16:31:24.608752 30755 raft_consensus.cc:3037] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:24.608819 30892 raft_consensus.cc:3037] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:24.609948 30755 raft_consensus.cc:2454] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7511a2044854916bdeb38c7078e48d9 in term 1.
I20260929 16:31:24.610004 30892 raft_consensus.cc:2454] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7511a2044854916bdeb38c7078e48d9 in term 1.
I20260929 16:31:24.610239 30964 leader_election.cc:302] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9b7e6cec6524c3082a32c7df0f4ab6c, e7511a2044854916bdeb38c7078e48d9; no voters:
I20260929 16:31:24.610396 31172 raft_consensus.cc:2781] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:24.610610 31172 raft_consensus.cc:686] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 1 LEADER]: Becoming Leader. State: Replica: e7511a2044854916bdeb38c7078e48d9, State: Running, Role: LEADER
I20260929 16:31:24.610855 31172 consensus_queue.cc:237] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } } peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } }
I20260929 16:31:24.611302 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.611069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.3:55151)
0929 16:31:24.611183 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.611238 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 16:31:24.613514 30438 catalog_manager.cc:5495] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7511a2044854916bdeb38c7078e48d9 (127.23.44.3). New cstate: current_term: 1 leader_uuid: "e7511a2044854916bdeb38c7078e48d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7511a2044854916bdeb38c7078e48d9" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 } health_report { overall_health: UNKNOWN } } }
I20260929 16:31:24.638487 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.634911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.23.44.3:45979)
0929 16:31:24.635074 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:24.635079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:24.635093 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:31:24.635264 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:24.635266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:24.635320 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:24.635441 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:24.635673 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.636199 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.637047 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:24.637294 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:24.637421 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:24.637669 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:24.638099 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:24.638218 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:24.638279 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 16:31:24.638329 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:31:24.641498 30755 raft_consensus.cc:1261] T 18432db9494943328ca1001d0ec426d7 P 30dfcbab3ef543848c69b4734509af52 [term 1 FOLLOWER]: Refusing update from remote peer e7511a2044854916bdeb38c7078e48d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:24.641670 30892 raft_consensus.cc:1261] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c [term 1 FOLLOWER]: Refusing update from remote peer e7511a2044854916bdeb38c7078e48d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:24.642097 31179 consensus_queue.cc:1035] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30dfcbab3ef543848c69b4734509af52" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:24.642284 31172 consensus_queue.cc:1035] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9b7e6cec6524c3082a32c7df0f4ab6c" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:24.644330 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.644046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.2:33883)
0929 16:31:24.644172 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.644234 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 16:31:24.645067 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.644674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.1:54307)
0929 16:31:24.644943 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.645000 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260929 16:31:24.648177 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.647924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.1:55145)
0929 16:31:24.648056 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.648106 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 16:31:24.648850 31185 mvcc.cc:205] Tried to move back new op lower bound from 7334705089085521920 to 7334705088968937472. Current Snapshot: MvccSnapshot[applied={T|T < 7334705089085521920}]
I20260929 16:31:24.649078 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.648592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.3:39243)
0929 16:31:24.648952 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.649009 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
I20260929 16:31:24.649387 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.649041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.2:42301)
0929 16:31:24.649239 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.649322 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260929 16:31:24.651528 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.651260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44913 (local address 127.23.44.3:38999)
0929 16:31:24.651418 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.651468 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 16:31:24.658177 30433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:24.658309 30433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:24.662078 30433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ad0e747da51470fbc4a8d4458498b64: Sending DeleteTablet for 3 replicas of tablet 18432db9494943328ca1001d0ec426d7
I20260929 16:31:24.663025 30872 tablet_service.cc:1514] Processing DeleteTablet for tablet 18432db9494943328ca1001d0ec426d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:24 UTC) from {username='oryx', principal='oryx/127.23.44.61@KRBTEST.COM'} at 127.0.0.1:56432
I20260929 16:31:24.663048 30735 tablet_service.cc:1514] Processing DeleteTablet for tablet 18432db9494943328ca1001d0ec426d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:24 UTC) from {username='oryx', principal='oryx/127.23.44.61@KRBTEST.COM'} at 127.0.0.1:59136
I20260929 16:31:24.663406 31193 tablet_replica.cc:330] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c: stopping tablet replica
I20260929 16:31:24.663463 31008 tablet_service.cc:1514] Processing DeleteTablet for tablet 18432db9494943328ca1001d0ec426d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:24 UTC) from {username='oryx', principal='oryx/127.23.44.61@KRBTEST.COM'} at 127.0.0.1:36612
I20260929 16:31:24.663540 31193 raft_consensus.cc:2229] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:24.663672 31193 raft_consensus.cc:2258] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:24.664143 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 30582
I20260929 16:31:24.666007 31193 ts_tablet_manager.cc:1905] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:24.666841 31195 tablet_replica.cc:330] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9: stopping tablet replica
I20260929 16:31:24.668303 31193 ts_tablet_manager.cc:1918] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:24.668375 31193 log.cc:1196] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/18432db9494943328ca1001d0ec426d7
I20260929 16:31:24.668612 31193 ts_tablet_manager.cc:1939] T 18432db9494943328ca1001d0ec426d7 P c9b7e6cec6524c3082a32c7df0f4ab6c: Deleting consensus metadata
I20260929 16:31:24.669406 31195 raft_consensus.cc:2229] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:24.669443 30425 catalog_manager.cc:4841] TS c9b7e6cec6524c3082a32c7df0f4ab6c (127.23.44.2:38115): tablet 18432db9494943328ca1001d0ec426d7 (table test-table [id=3e79126406ab46928ac10d5de3599e77]) successfully deleted
I20260929 16:31:24.669696 31195 raft_consensus.cc:2258] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:24.670403 31195 ts_tablet_manager.cc:1905] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:24.672999 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 30809
I20260929 16:31:24.673276 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:24.672929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:36317 (local address 127.0.0.1:59142)
0929 16:31:24.673090 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:24.673162 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 16:31:24.673779 31195 ts_tablet_manager.cc:1918] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:24.673908 31195 log.cc:1196] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/18432db9494943328ca1001d0ec426d7
I20260929 16:31:24.674235 31195 ts_tablet_manager.cc:1939] T 18432db9494943328ca1001d0ec426d7 P e7511a2044854916bdeb38c7078e48d9: Deleting consensus metadata
W20260929 16:31:24.674352 30425 catalog_manager.cc:4584] TS 30dfcbab3ef543848c69b4734509af52 (127.23.44.1:36317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18432db9494943328ca1001d0ec426d7: Network error: Client connection negotiation failed: client connection to 127.23.44.1:36317: connect: Connection refused (error 111)
I20260929 16:31:24.675094 30425 catalog_manager.cc:4841] TS e7511a2044854916bdeb38c7078e48d9 (127.23.44.3:45979): tablet 18432db9494943328ca1001d0ec426d7 (table test-table [id=3e79126406ab46928ac10d5de3599e77]) successfully deleted
I20260929 16:31:24.681216 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 30945
I20260929 16:31:24.686810 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 30302
I20260929 16:31:24.692725 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 30410
2026-09-29T16:31:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5524 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:24 dist-test-slave-mlmz krb5kdc[31198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:24 dist-test-slave-mlmz krb5kdc[31198](info): set up 2 sockets
Sep 29 16:31:24 dist-test-slave-mlmz krb5kdc[31198](info): commencing operation
krb5kdc: starting...
W20260929 16:31:25.786124 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:31:25 dist-test-slave-mlmz krb5kdc[31198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.62.keytab.
Entry for principal oryx/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.62.keytab.
I20260929 16:31:25.917832 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:40199
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.62.keytab
--principal=oryx/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:40199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:26.025599 31211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:26.025849 31211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:26.025889 31211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:26.029938 31211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:26.029992 31211 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:31:26.030009 31211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:26.030026 31211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:26.030042 31211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:26.030074 31211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:26.035455 31211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:40199
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.62.keytab
--principal=oryx/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:40199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:26.036517 31211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:26.037559 31211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:26.046259 31216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:26.046275 31217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:26.046275 31219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:26.046649 31211 server_base.cc:1034] running on GCE node
I20260929 16:31:26.047294 31211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:26.048406 31211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:26.049602 31211 hybrid_clock.cc:648] HybridClock initialized: now 1790699486049573 us; error 43 us; skew 500 ppm
Sep 29 16:31:26 dist-test-slave-mlmz krb5kdc[31198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:26.052464 31211 init.cc:377] Logged in from keytab as oryx/127.23.44.62@KRBTEST.COM (short username oryx)
I20260929 16:31:26.053450 31211 webserver.cc:460] Webserver started at http://127.23.44.62:45555/ using document root <none> and password file <none>
I20260929 16:31:26.053989 31211 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:26.054075 31211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:26.054291 31211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:26.056259 31211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "9f75b20fa5de4a268c1efd66287ca0b5"
format_stamp: "Formatted at 2026-09-29 16:31:26 on dist-test-slave-mlmz"
server_key: "fc56825ce5418b323a99ea4d436dcfec"
server_key_iv: "502413c4f6f62414d38b3e881a7c6af1"
server_key_version: "encryptionkey@0"
I20260929 16:31:26.056808 31211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "9f75b20fa5de4a268c1efd66287ca0b5"
format_stamp: "Formatted at 2026-09-29 16:31:26 on dist-test-slave-mlmz"
server_key: "fc56825ce5418b323a99ea4d436dcfec"
server_key_iv: "502413c4f6f62414d38b3e881a7c6af1"
server_key_version: "encryptionkey@0"
I20260929 16:31:26.060497 31211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260929 16:31:26.062881 31226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:26.064116 31211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:31:26.064250 31211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "9f75b20fa5de4a268c1efd66287ca0b5"
format_stamp: "Formatted at 2026-09-29 16:31:26 on dist-test-slave-mlmz"
server_key: "fc56825ce5418b323a99ea4d436dcfec"
server_key_iv: "502413c4f6f62414d38b3e881a7c6af1"
server_key_version: "encryptionkey@0"
I20260929 16:31:26.064358 31211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:26.075522 31211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:26.078624 31211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:26.078824 31211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:26.086615 31211 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:40199
I20260929 16:31:26.086673 31288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:40199 every 8 connection(s)
I20260929 16:31:26.087553 31211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:26.090752 31289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:26.093726 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 31211
I20260929 16:31:26.093842 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:26.094132 23728 external_mini_cluster.cc:1428] Setting key d67ca876cf6ba11810b3c0676947e5c6
I20260929 16:31:26.096935 31289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5: Bootstrap starting.
I20260929 16:31:26.099241 31289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:26.099998 31289 log.cc:824] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:26.102002 31289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5: No bootstrap required, opened a new log
Sep 29 16:31:26 dist-test-slave-mlmz krb5kdc[31198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
I20260929 16:31:26.106038 31289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f75b20fa5de4a268c1efd66287ca0b5" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40199 } }
I20260929 16:31:26.106458 31289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:26.106549 31289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f75b20fa5de4a268c1efd66287ca0b5, State: Initialized, Role: FOLLOWER
I20260929 16:31:26.107106 31289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f75b20fa5de4a268c1efd66287ca0b5" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40199 } }
I20260929 16:31:26.107270 31289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:26.107347 31289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:26.107451 31289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:26.108579 31289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f75b20fa5de4a268c1efd66287ca0b5" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40199 } }
I20260929 16:31:26.108923 31289 leader_election.cc:302] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f75b20fa5de4a268c1efd66287ca0b5; no voters:
I20260929 16:31:26.109033 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.095597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.23.44.62:40199)
0929 16:31:26.096069 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.096085 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.096135 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:31:26.096788 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.096804 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.097209 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.097625 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:26.098454 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.099625 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.100577 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.101181 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.101431 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.103885 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:26.103890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:26.103903 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:26.103928 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:26.105672 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.106186 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.106198 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.106202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.106311 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.106689 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.106692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.106694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.107207 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:26.107452 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.107783 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.108066 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":144,"threads_started":1}
I20260929 16:31:26.109232 31289 leader_election.cc:288] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:26.109419 31294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:26.109617 31294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9f75b20fa5de4a268c1efd66287ca0b5, State: Running, Role: LEADER
I20260929 16:31:26.109907 31294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f75b20fa5de4a268c1efd66287ca0b5" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40199 } }
I20260929 16:31:26.110388 31289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:26.111446 31295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f75b20fa5de4a268c1efd66287ca0b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f75b20fa5de4a268c1efd66287ca0b5" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40199 } } }
I20260929 16:31:26.111568 31295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:26.111796 31295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f75b20fa5de4a268c1efd66287ca0b5. Latest consensus state: current_term: 1 leader_uuid: "9f75b20fa5de4a268c1efd66287ca0b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f75b20fa5de4a268c1efd66287ca0b5" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40199 } } }
I20260929 16:31:26.111894 31295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:26.112360 31303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:26.115094 31303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:26.120092 31303 catalog_manager.cc:1347] Generated new cluster ID: b7e9a70c45054a2994d01267b7f75424
I20260929 16:31:26.120151 31303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:26.156364 31303 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:26.157521 31303 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:26.167150 31303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5: Generated new TSK 0
I20260929 16:31:26.167955 31303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab.
Entry for principal oryx/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab.
I20260929 16:31:26.200490 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:40199
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab
--principal=oryx/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:26.311139 31315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:26.311472 31315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:26.311559 31315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:26.316216 31315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:26.316270 31315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:26.316313 31315 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:31:26.316349 31315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:26.321776 31315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab
--principal=oryx/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.44.62:40199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:26.322662 31315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:26.323556 31315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:26.332837 31323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:26.332850 31320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:26.333009 31315 server_base.cc:1034] running on GCE node
W20260929 16:31:26.332842 31321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:26.334462 31315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:26.335011 31315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:26.336179 31315 hybrid_clock.cc:648] HybridClock initialized: now 1790699486336130 us; error 61 us; skew 500 ppm
Sep 29 16:31:26 dist-test-slave-mlmz krb5kdc[31198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:26.339069 31315 init.cc:377] Logged in from keytab as oryx/127.23.44.1@KRBTEST.COM (short username oryx)
I20260929 16:31:26.339916 31315 webserver.cc:460] Webserver started at http://127.23.44.1:36121/ using document root <none> and password file <none>
I20260929 16:31:26.340471 31315 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:26.340561 31315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:26.340785 31315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:26.342840 31315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "abe72324831b4807b6c8e466cbc3b20f"
format_stamp: "Formatted at 2026-09-29 16:31:26 on dist-test-slave-mlmz"
server_key: "236dc46ce80446e2885854d5ebeac79c"
server_key_iv: "64e8d6c21374defaa67d0d52fe6ba9d2"
server_key_version: "encryptionkey@0"
I20260929 16:31:26.343398 31315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "abe72324831b4807b6c8e466cbc3b20f"
format_stamp: "Formatted at 2026-09-29 16:31:26 on dist-test-slave-mlmz"
server_key: "236dc46ce80446e2885854d5ebeac79c"
server_key_iv: "64e8d6c21374defaa67d0d52fe6ba9d2"
server_key_version: "encryptionkey@0"
I20260929 16:31:26.347012 31315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:31:26.349929 31330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:26.351468 31315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:31:26.351624 31315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "abe72324831b4807b6c8e466cbc3b20f"
format_stamp: "Formatted at 2026-09-29 16:31:26 on dist-test-slave-mlmz"
server_key: "236dc46ce80446e2885854d5ebeac79c"
server_key_iv: "64e8d6c21374defaa67d0d52fe6ba9d2"
server_key_version: "encryptionkey@0"
I20260929 16:31:26.351733 31315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:26.370220 31315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:26.373340 31315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:26.373520 31315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:26.374822 31315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:26.374890 31315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:26.374954 31315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:26.375003 31315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:31:26 dist-test-slave-mlmz krb5kdc[31198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
I20260929 16:31:26.385514 31315 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:35253
I20260929 16:31:26.385587 31445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:35253 every 8 connection(s)
I20260929 16:31:26.386511 31315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:26.389681 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.375801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:53889 (local address 127.23.44.62:40199)
0929 16:31:26.375952 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.375956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.376914 (+ 958us) server_negotiation.cc:407] Connection header received
0929 16:31:26.378045 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.378049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.378116 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.378233 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:26.379921 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.380753 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.382631 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.382870 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.382955 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.385979 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:26.385983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:26.385986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:26.386012 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:26.387601 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.388179 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.388182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.388183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.388259 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.388603 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.388606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.388608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.388789 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:26.388896 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.389396 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.389514 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:31:26.390369 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.376241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40199 (local address 127.23.44.1:53889)
0929 16:31:26.376747 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:26.376788 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:26.377777 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:26.378459 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:26.378468 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:26.378974 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:26.379767 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.379789 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.380924 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.380929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.382492 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.382505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.383043 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.383047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.383262 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.383941 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:26.383969 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:26.385818 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:26.387721 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:26.387726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:26.387740 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:26.388060 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:26.388351 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:26.388355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:26.388357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:26.388499 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:26.388894 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:26.388901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:26.389125 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 16:31:26.389339 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 16:31:26.389632 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":131,"threads_started":1}
I20260929 16:31:26.391462 31446 heartbeater.cc:347] Connected to a master server at 127.23.44.62:40199
I20260929 16:31:26.391588 31446 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:26.392087 31446 heartbeater.cc:511] Master 127.23.44.62:40199 requested a full tablet report, sending...
I20260929 16:31:26.393581 31242 ts_manager.cc:194] Registered new tserver with Master: abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253)
I20260929 16:31:26.394682 31242 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:53889
I20260929 16:31:26.396786 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 31315
I20260929 16:31:26.396919 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:26.397231 23728 external_mini_cluster.cc:1428] Setting key 0947ee46c22e6cc8a2727effc1c0edb6
I20260929 16:31:26.402293 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.393754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:52061 (local address 127.23.44.62:40199)
0929 16:31:26.393869 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.393872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.395043 (+ 1171us) server_negotiation.cc:407] Connection header received
0929 16:31:26.395138 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.395140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.395183 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.395248 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:26.395745 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.396224 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.397334 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.397612 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.397749 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.399116 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:26.399119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:26.399121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:26.399141 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:26.400697 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.401134 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.401139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.401141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.401223 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.401532 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.401536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.401538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.401704 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:26.401813 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.402056 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.402156 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:31:26.402325 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.393634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40199 (local address 127.23.44.1:52061)
0929 16:31:26.394956 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:26.394970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:26.395062 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:26.395299 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:26.395302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:26.395504 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:26.395624 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.395638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.396349 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.396352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.397146 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.397159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.397762 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.397768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.397859 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.398499 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:26.398511 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:26.398958 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:26.400819 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:26.400824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:26.400826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:26.401025 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:26.401352 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:26.401356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:26.401358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:26.401438 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:26.401806 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:26.401809 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:26.401897 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 16:31:26.402004 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:31:26.402138 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.23.44.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab.
Entry for principal oryx/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab.
I20260929 16:31:26.428727 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:40199
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab
--principal=oryx/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:26.540784 31455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:26.541009 31455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:26.541052 31455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:26.545056 31455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:26.545107 31455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:26.545146 31455 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:31:26.545178 31455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:26.550335 31455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab
--principal=oryx/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.44.62:40199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:26.551617 31455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:26.552793 31455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:26.564044 31461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:26.564090 31463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:26.564095 31460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:26.564224 31455 server_base.cc:1034] running on GCE node
I20260929 16:31:26.564677 31455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:26.565407 31455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:26.566582 31455 hybrid_clock.cc:648] HybridClock initialized: now 1790699486566567 us; error 29 us; skew 500 ppm
Sep 29 16:31:26 dist-test-slave-mlmz krb5kdc[31198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:26.569629 31455 init.cc:377] Logged in from keytab as oryx/127.23.44.2@KRBTEST.COM (short username oryx)
I20260929 16:31:26.570506 31455 webserver.cc:460] Webserver started at http://127.23.44.2:45101/ using document root <none> and password file <none>
I20260929 16:31:26.571049 31455 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:26.571136 31455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:26.571359 31455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:26.573361 31455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "3e7fa8e3288945eeb18b766e9a1a0240"
format_stamp: "Formatted at 2026-09-29 16:31:26 on dist-test-slave-mlmz"
server_key: "02e560f15af2711fefa039efa3258582"
server_key_iv: "c5fa91764401fd7da01b84b9a9beb201"
server_key_version: "encryptionkey@0"
I20260929 16:31:26.573921 31455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "3e7fa8e3288945eeb18b766e9a1a0240"
format_stamp: "Formatted at 2026-09-29 16:31:26 on dist-test-slave-mlmz"
server_key: "02e560f15af2711fefa039efa3258582"
server_key_iv: "c5fa91764401fd7da01b84b9a9beb201"
server_key_version: "encryptionkey@0"
I20260929 16:31:26.577454 31455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:31:26.579967 31470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:26.581151 31455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:26.581313 31455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "3e7fa8e3288945eeb18b766e9a1a0240"
format_stamp: "Formatted at 2026-09-29 16:31:26 on dist-test-slave-mlmz"
server_key: "02e560f15af2711fefa039efa3258582"
server_key_iv: "c5fa91764401fd7da01b84b9a9beb201"
server_key_version: "encryptionkey@0"
I20260929 16:31:26.581426 31455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:26.599154 31455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:26.602196 31455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:26.602388 31455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:26.603818 31455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:26.603888 31455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:26.603957 31455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:26.604008 31455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:31:26 dist-test-slave-mlmz krb5kdc[31198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
I20260929 16:31:26.615134 31455 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:41643
I20260929 16:31:26.615154 31585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:41643 every 8 connection(s)
I20260929 16:31:26.616298 31455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:26.618856 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.604650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:43325 (local address 127.23.44.62:40199)
0929 16:31:26.604789 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.604793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.605800 (+ 1007us) server_negotiation.cc:407] Connection header received
0929 16:31:26.607020 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.607026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.607084 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.607135 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:26.608761 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.609510 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.611043 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.611488 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.611623 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.614838 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:26.614842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:26.614845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:26.614881 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:26.616601 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.617280 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.617286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.617288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.617382 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.617744 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.617749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.617751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.617948 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:26.618064 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.618553 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.618677 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:31:26.619594 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.605140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40199 (local address 127.23.44.2:43325)
0929 16:31:26.605651 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:26.605694 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:26.606728 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:26.607454 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:26.607460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:26.607876 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:26.608589 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.608614 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.609644 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.609647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.610903 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.610914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.611665 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.611671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.611912 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.612841 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:26.612866 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:26.614653 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:26.616727 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:26.616733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:26.616749 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:26.617132 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:26.617487 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:26.617492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:26.617494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:26.617636 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:26.618060 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:26.618066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:26.618300 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 16:31:26.618491 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 16:31:26.618821 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":133,"threads_started":1}
I20260929 16:31:26.620527 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 31455
I20260929 16:31:26.620648 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:26.620918 23728 external_mini_cluster.cc:1428] Setting key 28cf4adb70d85b35c58a13c5890fafa8
I20260929 16:31:26.621240 31586 heartbeater.cc:347] Connected to a master server at 127.23.44.62:40199
I20260929 16:31:26.621443 31586 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:26.622038 31586 heartbeater.cc:511] Master 127.23.44.62:40199 requested a full tablet report, sending...
I20260929 16:31:26.623077 31242 ts_manager.cc:194] Registered new tserver with Master: 3e7fa8e3288945eeb18b766e9a1a0240 (127.23.44.2:41643)
I20260929 16:31:26.623721 31242 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:43325
WARNING: no policy specified for oryx/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:31:26.631069 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.623372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:46635 (local address 127.23.44.62:40199)
0929 16:31:26.623504 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.623508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.623896 (+ 388us) server_negotiation.cc:407] Connection header received
0929 16:31:26.624021 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.624026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.624072 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.624122 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:26.624727 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.625214 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.626036 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.626260 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.626365 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.627458 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:26.627462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:26.627464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:26.627483 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:26.629226 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.629762 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.629767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.629769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.629857 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.630169 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.630172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.630174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.630409 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:26.630525 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.630799 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.630911 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:31:26.631088 31592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.623510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40199 (local address 127.23.44.2:46635)
0929 16:31:26.623789 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:26.623804 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:26.623904 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:26.624259 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:26.624262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:26.624450 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:26.624608 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.624615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.625353 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.625359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.625919 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.625926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.626367 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.626370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.626457 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.627000 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:26.627010 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:26.627311 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:26.629431 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:26.629435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:26.629438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:26.629651 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:26.629965 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:26.629969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:26.629971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:26.630060 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:26.630515 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:26.630518 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:26.630624 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 16:31:26.630742 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:31:26.630900 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab.
Entry for principal oryx/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab.
I20260929 16:31:26.649775 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:40199
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab
--principal=oryx/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:26.762534 31595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:26.762763 31595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:26.762806 31595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:26.766891 31595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:26.766940 31595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:26.767005 31595 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:31:26.767091 31595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:26.772312 31595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab
--principal=oryx/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.44.62:40199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:26.773320 31595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:26.774267 31595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:26.783908 31600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:26.784041 31595 server_base.cc:1034] running on GCE node
W20260929 16:31:26.783908 31603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:26.783946 31601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:26.784626 31595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:26.785413 31595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:26.786597 31595 hybrid_clock.cc:648] HybridClock initialized: now 1790699486786567 us; error 42 us; skew 500 ppm
Sep 29 16:31:26 dist-test-slave-mlmz krb5kdc[31198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:26.789419 31595 init.cc:377] Logged in from keytab as oryx/127.23.44.3@KRBTEST.COM (short username oryx)
I20260929 16:31:26.790380 31595 webserver.cc:460] Webserver started at http://127.23.44.3:36319/ using document root <none> and password file <none>
I20260929 16:31:26.790966 31595 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:26.791065 31595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:26.791355 31595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:26.793633 31595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "34a905ebcbd04cb3b6ee0a7812602eab"
format_stamp: "Formatted at 2026-09-29 16:31:26 on dist-test-slave-mlmz"
server_key: "df469c562317301e60fa5618d966713a"
server_key_iv: "2781bd8266b50bcc357d6c8d35fbaeef"
server_key_version: "encryptionkey@0"
I20260929 16:31:26.794229 31595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "34a905ebcbd04cb3b6ee0a7812602eab"
format_stamp: "Formatted at 2026-09-29 16:31:26 on dist-test-slave-mlmz"
server_key: "df469c562317301e60fa5618d966713a"
server_key_iv: "2781bd8266b50bcc357d6c8d35fbaeef"
server_key_version: "encryptionkey@0"
I20260929 16:31:26.798154 31595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:31:26.801054 31610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:26.802397 31595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:31:26.802551 31595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "34a905ebcbd04cb3b6ee0a7812602eab"
format_stamp: "Formatted at 2026-09-29 16:31:26 on dist-test-slave-mlmz"
server_key: "df469c562317301e60fa5618d966713a"
server_key_iv: "2781bd8266b50bcc357d6c8d35fbaeef"
server_key_version: "encryptionkey@0"
I20260929 16:31:26.802661 31595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:26.815547 31595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:26.818674 31595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:26.818881 31595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:26.820339 31595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:26.820416 31595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:26.820598 31595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:26.820624 31595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:31:26 dist-test-slave-mlmz krb5kdc[31198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
I20260929 16:31:26.832733 31595 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:40255
I20260929 16:31:26.832819 31725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:40255 every 8 connection(s)
I20260929 16:31:26.833961 31595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:26.835894 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 31595
I20260929 16:31:26.836004 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:26.836238 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.821107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:45559 (local address 127.23.44.62:40199)
0929 16:31:26.821312 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.821317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.822456 (+ 1139us) server_negotiation.cc:407] Connection header received
0929 16:31:26.823605 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.823609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.823671 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.823765 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:26.825190 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.826157 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.827482 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.827730 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.827858 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.831885 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:26.831889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:26.831892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:26.831930 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:26.833837 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.834414 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.834420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.834422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.834660 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.835102 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.835105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.835106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.835308 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:26.835397 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.835931 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.836059 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 16:31:26.836256 23728 external_mini_cluster.cc:1428] Setting key f56cb67c093d1a344ad07c32f34c5b10
I20260929 16:31:26.837086 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.821746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40199 (local address 127.23.44.3:45559)
0929 16:31:26.822235 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:26.822335 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:26.823372 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:26.824000 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:26.824007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:26.824545 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:26.825023 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.825037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.826376 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.826380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.827341 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.827351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.827897 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.827902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.828092 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.828800 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:26.828832 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:26.831691 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:26.833957 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:26.833962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:26.833972 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:26.834297 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:26.834796 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:26.834802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:26.834807 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:26.834986 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:26.835418 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:26.835423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:26.835645 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 16:31:26.835854 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 16:31:26.836178 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":143,"threads_started":1}
I20260929 16:31:26.838373 31726 heartbeater.cc:347] Connected to a master server at 127.23.44.62:40199
I20260929 16:31:26.838596 31726 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:26.839063 31726 heartbeater.cc:511] Master 127.23.44.62:40199 requested a full tablet report, sending...
I20260929 16:31:26.840055 31242 ts_manager.cc:194] Registered new tserver with Master: 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255)
I20260929 16:31:26.840607 31242 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:45559
I20260929 16:31:26.848178 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.840796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40199 (local address 127.23.44.3:50557)
0929 16:31:26.841095 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:26.841110 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:26.841200 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:26.841452 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:26.841454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:26.841612 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:26.841736 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.841742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.842809 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.842812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.843554 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.843565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.844020 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.844023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.844086 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.844561 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:26.844574 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:26.844892 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:26.846760 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:26.846762 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:26.846764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:26.846987 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:26.847292 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:26.847295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:26.847296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:26.847379 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:26.847759 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:26.847762 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:26.847838 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 16:31:26.847928 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:31:26.848031 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":125,"threads_started":1}
I20260929 16:31:26.848198 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.840836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:50557 (local address 127.23.44.62:40199)
0929 16:31:26.840975 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.840979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.841185 (+ 206us) server_negotiation.cc:407] Connection header received
0929 16:31:26.841304 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.841307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.841354 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.841443 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:26.841851 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.842653 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.843673 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.843919 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.844008 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.845027 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:26.845031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:26.845033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:26.845054 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:26.846614 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.847088 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.847090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.847092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.847166 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.847475 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.847477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.847479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.847640 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:26.847721 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.847980 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.848077 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:31:26.850301 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:31:26 dist-test-slave-mlmz krb5kdc[31198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699486, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:31:26 dist-test-slave-mlmz krb5kdc[31198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:26 dist-test-slave-mlmz krb5kdc[31198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:31:26.876333 31292 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:26.869379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.23.44.62:40199)
0929 16:31:26.869508 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.869513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.869536 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:31:26.869578 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.869580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.869621 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.869697 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:26.870380 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.870958 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.871843 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.872051 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.872139 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.876180 (+ 4041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34718: BlockingRecv error: recv got EOF from 127.0.0.1:34718 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 16:31:26 dist-test-slave-mlmz krb5kdc[31198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699486, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
I20260929 16:31:26.886937 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.877934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.23.44.62:40199)
0929 16:31:26.878047 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.878050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.878177 (+ 127us) server_negotiation.cc:407] Connection header received
0929 16:31:26.878277 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.878279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.878314 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.878400 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:26.879036 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.879543 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.880408 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.880643 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.880738 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.883599 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:26.883602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:26.883604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:26.883625 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:26.885104 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.885698 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.885702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.885703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.885801 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:26.886095 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:26.886097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:26.886099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:26.886277 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:26.886383 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.886558 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.886666 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:31:26.890754 31242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:26.893988 31242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:26.911114 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.904782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.23.44.2:41643)
0929 16:31:26.906006 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.906019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.906040 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:31:26.906111 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.906116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.906325 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.906488 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:26.907008 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.907789 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.909440 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.910307 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.910511 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.910687 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.910835 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.910982 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":110,"threads_started":1}
I20260929 16:31:26.911098 31749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.904511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:41643 (local address 127.0.0.1:54742)
0929 16:31:26.905326 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:26.905340 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:26.905426 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:26.906490 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:26.906496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:26.906564 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:26.906794 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.906802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.907901 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.907907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.909316 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.909326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.910510 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.910514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.910630 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.910678 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 16:31:26.910804 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 16:31:26.910929 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":177,"spinlock_wait_cycles":70656,"thread_start_us":120,"threads_started":1}
I20260929 16:31:26.911540 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.904509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:35253 (local address 127.0.0.1:36116)
0929 16:31:26.904907 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:26.904955 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:26.905072 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:26.906498 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:26.906501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:26.906554 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:26.906730 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.906740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.908389 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.908393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.909878 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.909892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.910839 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.910843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.911228 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.911259 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:31:26.911331 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 16:31:26.911387 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":368,"spinlock_wait_cycles":2048,"thread_start_us":103,"threads_started":1}
I20260929 16:31:26.911788 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.905310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:40255 (local address 127.0.0.1:51048)
0929 16:31:26.905637 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:26.905653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:26.905742 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:26.906498 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:26.906501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:26.906538 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:26.906801 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.906808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.908389 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.908393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.909679 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.909687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.910812 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.910819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.910941 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.910964 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:31:26.911033 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:31:26.911102 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":194,"spinlock_wait_cycles":115712,"thread_start_us":99,"threads_started":1}
I20260929 16:31:26.912475 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.905525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.23.44.1:35253)
0929 16:31:26.905978 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.905986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.906010 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:31:26.906093 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.906099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.906328 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.906894 (+ 566us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:26.907039 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.908252 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.910002 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.910725 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.910927 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.911123 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.912272 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.912369 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":87,"threads_started":1}
I20260929 16:31:26.912811 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.905593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.23.44.3:40255)
0929 16:31:26.906031 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.906043 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.906062 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:31:26.906167 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.906173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.906348 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.906664 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:26.906917 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.908240 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.909790 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.910663 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.911928 (+ 1265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.912097 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.912149 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.912699 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":106,"threads_started":1}
I20260929 16:31:26.914578 31520 tablet_service.cc:1467] Processing CreateTablet for tablet f9e413ff883c45a3ac9a27d85da037f2 (DEFAULT_TABLE table=test-table [id=39f07a139cde49ac88b3f4ed390517d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:26.914868 31660 tablet_service.cc:1467] Processing CreateTablet for tablet f9e413ff883c45a3ac9a27d85da037f2 (DEFAULT_TABLE table=test-table [id=39f07a139cde49ac88b3f4ed390517d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:26.915150 31380 tablet_service.cc:1467] Processing CreateTablet for tablet f9e413ff883c45a3ac9a27d85da037f2 (DEFAULT_TABLE table=test-table [id=39f07a139cde49ac88b3f4ed390517d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:26.915769 31520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e413ff883c45a3ac9a27d85da037f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:26.915856 31660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e413ff883c45a3ac9a27d85da037f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:26.916077 31380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e413ff883c45a3ac9a27d85da037f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:26.922369 31754 tablet_bootstrap.cc:492] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab: Bootstrap starting.
I20260929 16:31:26.924120 31756 tablet_bootstrap.cc:492] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240: Bootstrap starting.
I20260929 16:31:26.924268 31754 tablet_bootstrap.cc:654] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:26.924857 31755 tablet_bootstrap.cc:492] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f: Bootstrap starting.
I20260929 16:31:26.925122 31754 log.cc:824] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:26.926342 31756 tablet_bootstrap.cc:654] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:26.927019 31755 tablet_bootstrap.cc:654] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:26.927230 31754 tablet_bootstrap.cc:492] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab: No bootstrap required, opened a new log
I20260929 16:31:26.927220 31756 log.cc:824] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:26.927453 31754 ts_tablet_manager.cc:1397] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:31:26.927857 31755 log.cc:824] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:26.929148 31756 tablet_bootstrap.cc:492] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240: No bootstrap required, opened a new log
I20260929 16:31:26.929370 31756 ts_tablet_manager.cc:1397] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:31:26.929574 31755 tablet_bootstrap.cc:492] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f: No bootstrap required, opened a new log
I20260929 16:31:26.929768 31755 ts_tablet_manager.cc:1397] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:31:26.931306 31754 raft_consensus.cc:348] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:26.931561 31754 raft_consensus.cc:374] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:26.931614 31754 raft_consensus.cc:729] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a905ebcbd04cb3b6ee0a7812602eab, State: Initialized, Role: FOLLOWER
I20260929 16:31:26.932078 31754 consensus_queue.cc:260] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:26.933046 31754 ts_tablet_manager.cc:1428] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:31:26.933403 31726 heartbeater.cc:503] Master 127.23.44.62:40199 was elected leader, sending a full tablet report...
I20260929 16:31:26.933913 31755 raft_consensus.cc:348] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:26.934111 31755 raft_consensus.cc:374] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:26.934172 31755 raft_consensus.cc:729] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe72324831b4807b6c8e466cbc3b20f, State: Initialized, Role: FOLLOWER
I20260929 16:31:26.934374 31756 raft_consensus.cc:348] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:26.934613 31756 raft_consensus.cc:374] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:26.934641 31755 consensus_queue.cc:260] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:26.934667 31756 raft_consensus.cc:729] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7fa8e3288945eeb18b766e9a1a0240, State: Initialized, Role: FOLLOWER
I20260929 16:31:26.935258 31446 heartbeater.cc:503] Master 127.23.44.62:40199 was elected leader, sending a full tablet report...
I20260929 16:31:26.935262 31756 consensus_queue.cc:260] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:26.935647 31755 ts_tablet_manager.cc:1428] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:31:26.936162 31586 heartbeater.cc:503] Master 127.23.44.62:40199 was elected leader, sending a full tablet report...
I20260929 16:31:26.936195 31756 ts_tablet_manager.cc:1428] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 16:31:26.984655 31762 raft_consensus.cc:482] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:26.984894 31762 raft_consensus.cc:504] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:26.986330 31762 leader_election.cc:288] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253), 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255)
I20260929 16:31:26.991431 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.986766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:35253 (local address 127.23.44.2:36903)
0929 16:31:26.986961 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:26.986981 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:26.987087 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:26.987525 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:26.987529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:26.987551 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:26.987705 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.987720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.988827 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.988830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.989937 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.989946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.990942 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.990948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.991059 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.991084 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:31:26.991210 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 16:31:26.991285 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 16:31:26.991712 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.986901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:36903 (local address 127.23.44.1:35253)
0929 16:31:26.987192 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.987199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.987221 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:31:26.987280 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.987283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.987338 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.987817 (+ 479us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:26.987907 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.988663 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.990081 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.990766 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.990904 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.990997 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.991332 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.991471 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260929 16:31:26.992304 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.987317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:40255 (local address 127.23.44.2:58299)
0929 16:31:26.987674 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:26.987685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:26.987811 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:26.988094 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:26.988097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:26.988111 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:26.988573 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.988579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.989666 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.989669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.990826 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:26.990838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.991904 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.991909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:26.992037 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.992054 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 16:31:26.992104 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 16:31:26.992151 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":157,"threads_started":1}
I20260929 16:31:26.992532 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e413ff883c45a3ac9a27d85da037f2" candidate_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe72324831b4807b6c8e466cbc3b20f" is_pre_election: true
I20260929 16:31:26.992667 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:26.987459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:58299 (local address 127.23.44.3:40255)
0929 16:31:26.987587 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:26.987591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:26.987772 (+ 181us) server_negotiation.cc:407] Connection header received
0929 16:31:26.987929 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:26.987932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:26.987983 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:26.988074 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:26.988697 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.989524 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.990964 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:26.991545 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:26.991706 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:26.991771 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:26.992470 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 16:31:26.992547 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:31:26.992866 31400 raft_consensus.cc:2454] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e7fa8e3288945eeb18b766e9a1a0240 in term 0.
I20260929 16:31:26.993459 31473 leader_election.cc:302] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e7fa8e3288945eeb18b766e9a1a0240, abe72324831b4807b6c8e466cbc3b20f; no voters:
I20260929 16:31:26.993762 31762 raft_consensus.cc:2781] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:26.993877 31762 raft_consensus.cc:482] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:26.993937 31762 raft_consensus.cc:3037] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:26.993583 31680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e413ff883c45a3ac9a27d85da037f2" candidate_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" is_pre_election: true
I20260929 16:31:26.994402 31680 raft_consensus.cc:2454] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e7fa8e3288945eeb18b766e9a1a0240 in term 0.
I20260929 16:31:26.995175 31762 raft_consensus.cc:504] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:26.995711 31762 leader_election.cc:288] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [CANDIDATE]: Term 1 election: Requested vote from peers abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253), 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255)
I20260929 16:31:26.996251 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e413ff883c45a3ac9a27d85da037f2" candidate_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe72324831b4807b6c8e466cbc3b20f"
I20260929 16:31:26.996460 31680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e413ff883c45a3ac9a27d85da037f2" candidate_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a905ebcbd04cb3b6ee0a7812602eab"
I20260929 16:31:26.996510 31400 raft_consensus.cc:3037] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:26.996621 31680 raft_consensus.cc:3037] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:26.997961 31400 raft_consensus.cc:2454] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e7fa8e3288945eeb18b766e9a1a0240 in term 1.
I20260929 16:31:26.998322 31680 raft_consensus.cc:2454] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e7fa8e3288945eeb18b766e9a1a0240 in term 1.
I20260929 16:31:26.998401 31473 leader_election.cc:302] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e7fa8e3288945eeb18b766e9a1a0240, abe72324831b4807b6c8e466cbc3b20f; no voters:
I20260929 16:31:26.998677 31762 raft_consensus.cc:2781] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:26.998888 31762 raft_consensus.cc:686] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 LEADER]: Becoming Leader. State: Replica: 3e7fa8e3288945eeb18b766e9a1a0240, State: Running, Role: LEADER
I20260929 16:31:26.999248 31762 consensus_queue.cc:237] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.003714 31242 catalog_manager.cc:5495] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e7fa8e3288945eeb18b766e9a1a0240 (127.23.44.2). New cstate: current_term: 1 leader_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } health_report { overall_health: UNKNOWN } } }
I20260929 16:31:27.019217 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.014267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.23.44.2:41643)
0929 16:31:27.014424 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:27.014428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:27.014440 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:31:27.014513 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:27.014515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:27.014570 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:27.014694 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:27.015008 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.015557 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.016807 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.017109 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.017306 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.017679 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:27.018538 (+ 859us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:27.018798 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:27.018917 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 16:31:27.019015 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:31:27.027128 31400 raft_consensus.cc:1261] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Refusing update from remote peer 3e7fa8e3288945eeb18b766e9a1a0240: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:27.027128 31680 raft_consensus.cc:1261] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Refusing update from remote peer 3e7fa8e3288945eeb18b766e9a1a0240: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:27.027832 31762 consensus_queue.cc:1035] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:27.028034 31764 consensus_queue.cc:1035] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:27.036329 31769 mvcc.cc:205] Tried to move back new op lower bound from 7334705098855202816 to 7334705098754101248. Current Snapshot: MvccSnapshot[applied={T|T < 7334705098855202816}]
I20260929 16:31:27.038893 31768 mvcc.cc:205] Tried to move back new op lower bound from 7334705098855202816 to 7334705098754101248. Current Snapshot: MvccSnapshot[applied={T|T < 7334705098855202816 or (T in {7334705098855202816})}]
I20260929 16:31:27.044600 31242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:27.044850 31242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:27.047853 31242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5: Sending DeleteTablet for 3 replicas of tablet f9e413ff883c45a3ac9a27d85da037f2
I20260929 16:31:27.048796 31660 tablet_service.cc:1514] Processing DeleteTablet for tablet f9e413ff883c45a3ac9a27d85da037f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:27 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260929 16:31:27.049034 31380 tablet_service.cc:1514] Processing DeleteTablet for tablet f9e413ff883c45a3ac9a27d85da037f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:27 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:36116
I20260929 16:31:27.049034 31520 tablet_service.cc:1514] Processing DeleteTablet for tablet f9e413ff883c45a3ac9a27d85da037f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:27 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:54742
I20260929 16:31:27.049541 31777 tablet_replica.cc:330] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab: stopping tablet replica
I20260929 16:31:27.049906 31777 raft_consensus.cc:2229] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:27.050261 31777 raft_consensus.cc:2258] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:27.050310 31778 tablet_replica.cc:330] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240: stopping tablet replica
I20260929 16:31:27.050617 31778 raft_consensus.cc:2229] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:27.050534 31242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:27.050932 31242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:27.050998 31778 raft_consensus.cc:2258] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:27.051405 31779 tablet_replica.cc:330] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f: stopping tablet replica
I20260929 16:31:27.051704 31779 raft_consensus.cc:2229] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:27.051982 31777 ts_tablet_manager.cc:1905] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:27.051998 31779 raft_consensus.cc:2258] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:27.054018 31778 ts_tablet_manager.cc:1905] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:27.056663 31520 tablet_service.cc:1467] Processing CreateTablet for tablet 928d882b98294d298759dccd1a6c31f9 (DEFAULT_TABLE table=test-table [id=0a988b9603a949cd88749ea12dac5b05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:27.057166 31777 ts_tablet_manager.cc:1918] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:27.057301 31777 log.cc:1196] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/f9e413ff883c45a3ac9a27d85da037f2
I20260929 16:31:27.057379 31380 tablet_service.cc:1467] Processing CreateTablet for tablet 928d882b98294d298759dccd1a6c31f9 (DEFAULT_TABLE table=test-table [id=0a988b9603a949cd88749ea12dac5b05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:27.057430 31660 tablet_service.cc:1467] Processing CreateTablet for tablet 928d882b98294d298759dccd1a6c31f9 (DEFAULT_TABLE table=test-table [id=0a988b9603a949cd88749ea12dac5b05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:27.057636 31380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928d882b98294d298759dccd1a6c31f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:27.057636 31660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928d882b98294d298759dccd1a6c31f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:27.059567 31755 tablet_bootstrap.cc:492] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f: Bootstrap starting.
I20260929 16:31:27.060073 31754 tablet_bootstrap.cc:492] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab: Bootstrap starting.
I20260929 16:31:27.060287 31520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928d882b98294d298759dccd1a6c31f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:27.060550 31755 tablet_bootstrap.cc:654] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:27.061059 31754 tablet_bootstrap.cc:654] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:27.062249 31754 tablet_bootstrap.cc:492] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab: No bootstrap required, opened a new log
I20260929 16:31:27.062327 31754 ts_tablet_manager.cc:1397] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:31:27.057636 31777 ts_tablet_manager.cc:1939] T f9e413ff883c45a3ac9a27d85da037f2 P 34a905ebcbd04cb3b6ee0a7812602eab: Deleting consensus metadata
I20260929 16:31:27.062695 31756 tablet_bootstrap.cc:492] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240: Bootstrap starting.
I20260929 16:31:27.062800 31754 raft_consensus.cc:348] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.062901 31754 raft_consensus.cc:374] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:27.062933 31754 raft_consensus.cc:729] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a905ebcbd04cb3b6ee0a7812602eab, State: Initialized, Role: FOLLOWER
I20260929 16:31:27.063053 31754 consensus_queue.cc:260] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.063321 31754 ts_tablet_manager.cc:1428] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 16:31:27.063632 31756 tablet_bootstrap.cc:654] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:27.064791 31755 tablet_bootstrap.cc:492] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f: No bootstrap required, opened a new log
I20260929 16:31:27.064877 31755 ts_tablet_manager.cc:1397] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 16:31:27.064817 31229 catalog_manager.cc:4841] TS 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255): tablet f9e413ff883c45a3ac9a27d85da037f2 (table test-table [id=39f07a139cde49ac88b3f4ed390517d5]) successfully deleted
I20260929 16:31:27.065333 31755 raft_consensus.cc:348] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.065444 31755 raft_consensus.cc:374] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:27.065478 31755 raft_consensus.cc:729] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe72324831b4807b6c8e466cbc3b20f, State: Initialized, Role: FOLLOWER
I20260929 16:31:27.065584 31755 consensus_queue.cc:260] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.065816 31755 ts_tablet_manager.cc:1428] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:27.066622 31778 ts_tablet_manager.cc:1918] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:27.066789 31778 log.cc:1196] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/f9e413ff883c45a3ac9a27d85da037f2
I20260929 16:31:27.065824 31779 ts_tablet_manager.cc:1905] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:27.067214 31778 ts_tablet_manager.cc:1939] T f9e413ff883c45a3ac9a27d85da037f2 P 3e7fa8e3288945eeb18b766e9a1a0240: Deleting consensus metadata
I20260929 16:31:27.068526 31230 catalog_manager.cc:4841] TS 3e7fa8e3288945eeb18b766e9a1a0240 (127.23.44.2:41643): tablet f9e413ff883c45a3ac9a27d85da037f2 (table test-table [id=39f07a139cde49ac88b3f4ed390517d5]) successfully deleted
I20260929 16:31:27.068987 31756 tablet_bootstrap.cc:492] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240: No bootstrap required, opened a new log
I20260929 16:31:27.069090 31756 ts_tablet_manager.cc:1397] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260929 16:31:27.069619 31756 raft_consensus.cc:348] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.069761 31756 raft_consensus.cc:374] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:27.069800 31756 raft_consensus.cc:729] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7fa8e3288945eeb18b766e9a1a0240, State: Initialized, Role: FOLLOWER
I20260929 16:31:27.069944 31756 consensus_queue.cc:260] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.070197 31756 ts_tablet_manager.cc:1428] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:27.071012 31779 ts_tablet_manager.cc:1918] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:27.071107 31779 log.cc:1196] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/wal/wals/f9e413ff883c45a3ac9a27d85da037f2
I20260929 16:31:27.071416 31779 ts_tablet_manager.cc:1939] T f9e413ff883c45a3ac9a27d85da037f2 P abe72324831b4807b6c8e466cbc3b20f: Deleting consensus metadata
I20260929 16:31:27.072538 31229 catalog_manager.cc:4841] TS abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253): tablet f9e413ff883c45a3ac9a27d85da037f2 (table test-table [id=39f07a139cde49ac88b3f4ed390517d5]) successfully deleted
W20260929 16:31:27.085409 31727 tablet.cc:2369] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:31:27.117463 31587 tablet.cc:2369] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:31:27.138616 31447 tablet.cc:2369] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:31:27.208738 31762 raft_consensus.cc:482] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:27.208871 31762 raft_consensus.cc:504] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.209331 31762 leader_election.cc:288] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253), 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255)
I20260929 16:31:27.209841 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928d882b98294d298759dccd1a6c31f9" candidate_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe72324831b4807b6c8e466cbc3b20f" is_pre_election: true
I20260929 16:31:27.209883 31680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928d882b98294d298759dccd1a6c31f9" candidate_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" is_pre_election: true
I20260929 16:31:27.210065 31400 raft_consensus.cc:2454] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e7fa8e3288945eeb18b766e9a1a0240 in term 0.
I20260929 16:31:27.210063 31680 raft_consensus.cc:2454] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e7fa8e3288945eeb18b766e9a1a0240 in term 0.
I20260929 16:31:27.210390 31473 leader_election.cc:302] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a905ebcbd04cb3b6ee0a7812602eab, 3e7fa8e3288945eeb18b766e9a1a0240; no voters:
I20260929 16:31:27.210554 31762 raft_consensus.cc:2781] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:27.210613 31762 raft_consensus.cc:482] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:27.210654 31762 raft_consensus.cc:3037] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:27.211855 31762 raft_consensus.cc:504] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.212195 31762 leader_election.cc:288] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [CANDIDATE]: Term 1 election: Requested vote from peers abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253), 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255)
I20260929 16:31:27.212608 31680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928d882b98294d298759dccd1a6c31f9" candidate_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a905ebcbd04cb3b6ee0a7812602eab"
I20260929 16:31:27.212643 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928d882b98294d298759dccd1a6c31f9" candidate_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe72324831b4807b6c8e466cbc3b20f"
I20260929 16:31:27.212734 31680 raft_consensus.cc:3037] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:27.212747 31400 raft_consensus.cc:3037] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:27.213666 31400 raft_consensus.cc:2454] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e7fa8e3288945eeb18b766e9a1a0240 in term 1.
I20260929 16:31:27.213749 31680 raft_consensus.cc:2454] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e7fa8e3288945eeb18b766e9a1a0240 in term 1.
I20260929 16:31:27.213945 31473 leader_election.cc:302] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e7fa8e3288945eeb18b766e9a1a0240, abe72324831b4807b6c8e466cbc3b20f; no voters:
I20260929 16:31:27.214107 31762 raft_consensus.cc:2781] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:27.214190 31762 raft_consensus.cc:686] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 LEADER]: Becoming Leader. State: Replica: 3e7fa8e3288945eeb18b766e9a1a0240, State: Running, Role: LEADER
I20260929 16:31:27.214344 31762 consensus_queue.cc:237] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.215840 31243 catalog_manager.cc:5495] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e7fa8e3288945eeb18b766e9a1a0240 (127.23.44.2). New cstate: current_term: 1 leader_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } health_report { overall_health: UNKNOWN } } }
Sep 29 16:31:27 dist-test-slave-mlmz krb5kdc[31198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.62@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
I20260929 16:31:27.283070 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.275009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.23.44.62:40199)
0929 16:31:27.275140 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:27.275144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:27.275339 (+ 195us) server_negotiation.cc:407] Connection header received
0929 16:31:27.275445 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:27.275447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:27.275485 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:27.275563 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:27.276279 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.276796 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.277674 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.277907 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.278009 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.280107 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:27.280110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:27.280112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:27.280132 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:27.281625 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:27.282042 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:27.282044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:27.282046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:27.282116 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:27.282355 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:27.282357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:27.282358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:27.282513 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:27.282585 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:27.282850 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 16:31:27.282937 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:31:27.283074 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.274978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40199 (local address 127.0.0.1:34736)
0929 16:31:27.275249 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:27.275269 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:27.275369 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:27.275615 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:27.275618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:27.276012 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:27.276119 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.276125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.276893 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.276896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.277519 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.277527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.278026 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.278029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.278086 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.278592 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:27.278603 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:27.279969 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:27.281719 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:27.281727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:27.281732 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:27.281957 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:27.282199 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:27.282201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:27.282202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:27.282275 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:27.282603 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:27.282608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:27.282704 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 16:31:27.282797 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 16:31:27.282893 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":108,"threads_started":1}
I20260929 16:31:27.287128 31243 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:34736:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:31:27.287745 31243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:27.292832 31380 tablet_service.cc:1467] Processing CreateTablet for tablet 1f113ed317234d828cd102cf4f01127b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e5014ccddec48a88d83c99f0f2d4cfa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:31:27.292940 31660 tablet_service.cc:1467] Processing CreateTablet for tablet 1f113ed317234d828cd102cf4f01127b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e5014ccddec48a88d83c99f0f2d4cfa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:31:27.292941 31520 tablet_service.cc:1467] Processing CreateTablet for tablet 1f113ed317234d828cd102cf4f01127b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e5014ccddec48a88d83c99f0f2d4cfa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:31:27.293288 31660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f113ed317234d828cd102cf4f01127b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:27.293293 31380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f113ed317234d828cd102cf4f01127b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:27.293376 31520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f113ed317234d828cd102cf4f01127b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:27.295501 31756 tablet_bootstrap.cc:492] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240: Bootstrap starting.
I20260929 16:31:27.296074 31755 tablet_bootstrap.cc:492] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f: Bootstrap starting.
I20260929 16:31:27.296345 31756 tablet_bootstrap.cc:654] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:27.297026 31755 tablet_bootstrap.cc:654] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:27.297410 31754 tablet_bootstrap.cc:492] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: Bootstrap starting.
I20260929 16:31:27.298162 31755 tablet_bootstrap.cc:492] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f: No bootstrap required, opened a new log
I20260929 16:31:27.298220 31754 tablet_bootstrap.cc:654] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:27.298270 31755 ts_tablet_manager.cc:1397] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:27.298686 31755 raft_consensus.cc:348] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.298825 31755 raft_consensus.cc:374] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:27.298872 31755 raft_consensus.cc:729] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe72324831b4807b6c8e466cbc3b20f, State: Initialized, Role: FOLLOWER
I20260929 16:31:27.299005 31755 consensus_queue.cc:260] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.299311 31755 ts_tablet_manager.cc:1428] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:27.299495 31756 tablet_bootstrap.cc:492] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240: No bootstrap required, opened a new log
I20260929 16:31:27.299576 31756 ts_tablet_manager.cc:1397] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 16:31:27.299633 31761 tablet_replica.cc:439] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } } }
I20260929 16:31:27.299759 31761 tablet_replica.cc:442] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:27.300083 31756 raft_consensus.cc:348] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.300246 31756 raft_consensus.cc:374] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:27.300289 31756 raft_consensus.cc:729] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7fa8e3288945eeb18b766e9a1a0240, State: Initialized, Role: FOLLOWER
I20260929 16:31:27.300421 31756 consensus_queue.cc:260] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.300678 31756 ts_tablet_manager.cc:1428] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:31:27.300635 31762 tablet_replica.cc:439] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } } }
I20260929 16:31:27.300755 31762 tablet_replica.cc:442] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:27.301302 31754 tablet_bootstrap.cc:492] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: No bootstrap required, opened a new log
I20260929 16:31:27.301402 31754 ts_tablet_manager.cc:1397] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260929 16:31:27.301883 31754 raft_consensus.cc:348] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.302006 31754 raft_consensus.cc:374] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:27.302043 31754 raft_consensus.cc:729] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a905ebcbd04cb3b6ee0a7812602eab, State: Initialized, Role: FOLLOWER
I20260929 16:31:27.302165 31754 consensus_queue.cc:260] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.302443 31754 ts_tablet_manager.cc:1428] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:31:27.302394 31760 tablet_replica.cc:439] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } } }
I20260929 16:31:27.302516 31760 tablet_replica.cc:442] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:27.501398 31760 raft_consensus.cc:482] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:27.501564 31760 raft_consensus.cc:504] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.502729 31760 leader_election.cc:288] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253), 3e7fa8e3288945eeb18b766e9a1a0240 (127.23.44.2:41643)
I20260929 16:31:27.507042 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.503063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:60821 (local address 127.23.44.2:41643)
0929 16:31:27.503255 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:27.503262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:27.503348 (+ 86us) server_negotiation.cc:407] Connection header received
0929 16:31:27.503502 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:27.503505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:27.503552 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:27.503630 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:27.503923 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.504511 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.505641 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.506258 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.506375 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.506441 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:27.506798 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 16:31:27.506930 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 16:31:27.507045 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.503017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:43725 (local address 127.23.44.1:35253)
0929 16:31:27.503310 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:27.503314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:27.503329 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:31:27.503489 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:27.503492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:27.503532 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:27.503621 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:27.503908 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.504444 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.505641 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.506299 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.506393 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.506442 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:27.506798 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 16:31:27.506888 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260929 16:31:27.507091 31793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.502941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:35253 (local address 127.23.44.3:43725)
0929 16:31:27.503240 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:27.503255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:27.503362 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:27.503632 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:27.503634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:27.503649 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:27.503768 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.503806 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.504550 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.504557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.505513 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.505523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.506384 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.506388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.506574 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.506667 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 16:31:27.506742 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 16:31:27.506948 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":196,"thread_start_us":82,"threads_started":1}
I20260929 16:31:27.507107 31794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.502959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:41643 (local address 127.23.44.3:60821)
0929 16:31:27.503251 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:27.503265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:27.503387 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:27.503651 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:27.503654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:27.503697 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:27.503804 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.503810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.504644 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.504650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.505521 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.505527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.506384 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.506388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.506579 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.506663 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 16:31:27.506742 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:31:27.506910 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":141,"spinlock_wait_cycles":1408,"thread_start_us":73,"threads_started":1}
I20260929 16:31:27.507550 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f113ed317234d828cd102cf4f01127b" candidate_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe72324831b4807b6c8e466cbc3b20f" is_pre_election: true
I20260929 16:31:27.507686 31400 raft_consensus.cc:2454] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a905ebcbd04cb3b6ee0a7812602eab in term 0.
I20260929 16:31:27.507896 31539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f113ed317234d828cd102cf4f01127b" candidate_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" is_pre_election: true
I20260929 16:31:27.508029 31613 leader_election.cc:302] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a905ebcbd04cb3b6ee0a7812602eab, abe72324831b4807b6c8e466cbc3b20f; no voters:
I20260929 16:31:27.508131 31539 raft_consensus.cc:2454] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a905ebcbd04cb3b6ee0a7812602eab in term 0.
I20260929 16:31:27.508322 31760 raft_consensus.cc:2781] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:27.508384 31760 raft_consensus.cc:482] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:27.508435 31760 raft_consensus.cc:3037] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:27.509498 31760 raft_consensus.cc:504] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.509837 31760 leader_election.cc:288] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [CANDIDATE]: Term 1 election: Requested vote from peers abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253), 3e7fa8e3288945eeb18b766e9a1a0240 (127.23.44.2:41643)
I20260929 16:31:27.510125 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f113ed317234d828cd102cf4f01127b" candidate_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe72324831b4807b6c8e466cbc3b20f"
I20260929 16:31:27.510243 31400 raft_consensus.cc:3037] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:27.510291 31539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f113ed317234d828cd102cf4f01127b" candidate_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e7fa8e3288945eeb18b766e9a1a0240"
I20260929 16:31:27.510421 31539 raft_consensus.cc:3037] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:27.511175 31400 raft_consensus.cc:2454] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a905ebcbd04cb3b6ee0a7812602eab in term 1.
I20260929 16:31:27.511437 31613 leader_election.cc:302] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a905ebcbd04cb3b6ee0a7812602eab, abe72324831b4807b6c8e466cbc3b20f; no voters:
I20260929 16:31:27.511683 31760 raft_consensus.cc:2781] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:27.511831 31539 raft_consensus.cc:2454] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a905ebcbd04cb3b6ee0a7812602eab in term 1.
I20260929 16:31:27.511907 31760 raft_consensus.cc:686] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 LEADER]: Becoming Leader. State: Replica: 34a905ebcbd04cb3b6ee0a7812602eab, State: Running, Role: LEADER
I20260929 16:31:27.512162 31760 consensus_queue.cc:237] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.513386 31796 tablet_replica.cc:439] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: TxnStatusTablet state changed. Reason: New leader 34a905ebcbd04cb3b6ee0a7812602eab. Latest consensus state: current_term: 1 leader_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } } }
I20260929 16:31:27.513517 31796 tablet_replica.cc:442] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:31:27.514037 31798 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:31:27.514219 31798 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:31:27.514781 31241 catalog_manager.cc:5495] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab reported cstate change: term changed from 0 to 1, leader changed from <none> to 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3). New cstate: current_term: 1 leader_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } health_report { overall_health: HEALTHY } } }
I20260929 16:31:27.538610 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.533867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:40255 (local address 127.0.0.1:51064)
0929 16:31:27.534010 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:27.534028 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:27.534120 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:27.534580 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:27.534583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:27.534593 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:31:27.534684 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.534689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.535882 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.535884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.536700 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.536709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.537369 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.537372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.537442 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.537488 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:31:27.538341 (+ 853us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:31:27.538356 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 16:31:27.538421 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 16:31:27.538495 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 16:31:27.538718 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.533983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51064 (local address 127.23.44.3:40255)
0929 16:31:27.534235 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:27.534244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:27.534259 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:31:27.534333 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:27.534337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:27.534398 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:27.534520 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:27.534810 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.535731 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.536818 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.537234 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.537376 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.537634 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:27.538215 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:27.538338 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:27.538456 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 16:31:27.538519 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":89,"threads_started":1}
I20260929 16:31:27.542057 31400 raft_consensus.cc:1261] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Refusing update from remote peer 34a905ebcbd04cb3b6ee0a7812602eab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:27.542261 31539 raft_consensus.cc:1261] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Refusing update from remote peer 34a905ebcbd04cb3b6ee0a7812602eab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:27.542613 31796 consensus_queue.cc:1035] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:27.542922 31760 consensus_queue.cc:1035] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:27.544621 31762 tablet_replica.cc:439] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240: TxnStatusTablet state changed. Reason: New leader 34a905ebcbd04cb3b6ee0a7812602eab. Latest consensus state: current_term: 1 leader_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } } }
I20260929 16:31:27.544735 31762 tablet_replica.cc:442] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:27.544692 31761 tablet_replica.cc:439] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f: TxnStatusTablet state changed. Reason: New leader 34a905ebcbd04cb3b6ee0a7812602eab. Latest consensus state: current_term: 1 leader_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } } }
I20260929 16:31:27.544800 31761 tablet_replica.cc:442] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:27.546852 31760 tablet_replica.cc:439] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } } }
I20260929 16:31:27.546988 31760 tablet_replica.cc:442] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:31:27.548199 31760 tablet_replica.cc:439] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } } }
I20260929 16:31:27.548308 31760 tablet_replica.cc:442] T 1f113ed317234d828cd102cf4f01127b P 34a905ebcbd04cb3b6ee0a7812602eab: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:31:27.549064 31761 tablet_replica.cc:439] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } } }
I20260929 16:31:27.549162 31761 tablet_replica.cc:442] T 1f113ed317234d828cd102cf4f01127b P abe72324831b4807b6c8e466cbc3b20f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:27.550551 31762 tablet_replica.cc:439] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } } }
I20260929 16:31:27.550663 31762 tablet_replica.cc:442] T 1f113ed317234d828cd102cf4f01127b P 3e7fa8e3288945eeb18b766e9a1a0240: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:31:27.563223 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.559012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:39503 (local address 127.23.44.3:40255)
0929 16:31:27.559127 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:27.559132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:27.559503 (+ 371us) server_negotiation.cc:407] Connection header received
0929 16:31:27.559601 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:27.559604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:27.559673 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:27.559785 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:27.560128 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.560694 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.562132 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.562373 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.562466 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.562792 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:27.562871 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:27.562932 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:27.563060 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 16:31:27.563114 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 16:31:27.563288 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.558866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:40255 (local address 127.23.44.2:39503)
0929 16:31:27.559313 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:27.559329 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:27.559470 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:27.559818 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:27.559821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:27.559830 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:31:27.559944 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.559951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.560835 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.560838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.561964 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.561976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.562489 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.562493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.562589 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.562652 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:31:27.562957 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:31:27.562964 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:31:27.563037 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 16:31:27.563128 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":134,"threads_started":1}
I20260929 16:31:27.570029 31400 raft_consensus.cc:1261] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Refusing update from remote peer 3e7fa8e3288945eeb18b766e9a1a0240: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:27.570040 31680 raft_consensus.cc:1261] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Refusing update from remote peer 3e7fa8e3288945eeb18b766e9a1a0240: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:27.570423 31773 consensus_queue.cc:1035] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:27.570569 31762 consensus_queue.cc:1035] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:27.574224 31804 mvcc.cc:205] Tried to move back new op lower bound from 7334705101082378240 to 7334705099631300608. Current Snapshot: MvccSnapshot[applied={T|T < 7334705101082378240}]
I20260929 16:31:27.596132 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.591743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:35261 (local address 127.23.44.2:41643)
0929 16:31:27.591867 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:27.591872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:27.592130 (+ 258us) server_negotiation.cc:407] Connection header received
0929 16:31:27.592313 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:27.592318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:27.592396 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:27.592480 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:27.592803 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.593557 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.594781 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.595010 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.595098 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.595539 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:27.595624 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:27.595710 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:27.595954 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 16:31:27.596014 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:31:27.596360 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.591676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:41643 (local address 127.23.44.3:35261)
0929 16:31:27.592038 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:27.592051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:27.592180 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:27.592503 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:27.592506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:27.592515 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:31:27.592657 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.592663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.593662 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.593666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.594655 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.594666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.595236 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.595240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.595346 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.595394 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:31:27.595838 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:31:27.595845 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:31:27.595908 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 16:31:27.596184 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":116,"threads_started":1}
I20260929 16:31:27.619778 31243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:27.619949 31243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:27.622133 31243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5: Sending DeleteTablet for 3 replicas of tablet 928d882b98294d298759dccd1a6c31f9
I20260929 16:31:27.622820 31380 tablet_service.cc:1514] Processing DeleteTablet for tablet 928d882b98294d298759dccd1a6c31f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:27 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:36116
I20260929 16:31:27.623121 31520 tablet_service.cc:1514] Processing DeleteTablet for tablet 928d882b98294d298759dccd1a6c31f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:27 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:54742
I20260929 16:31:27.623149 31820 tablet_replica.cc:330] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f: stopping tablet replica
I20260929 16:31:27.623376 31820 raft_consensus.cc:2229] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:27.623528 31820 raft_consensus.cc:2258] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:27.624246 31820 ts_tablet_manager.cc:1905] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:27.626077 31660 tablet_service.cc:1514] Processing DeleteTablet for tablet 928d882b98294d298759dccd1a6c31f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:27 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260929 16:31:27.627465 31820 ts_tablet_manager.cc:1918] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:31:27.627591 31820 log.cc:1196] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/wal/wals/928d882b98294d298759dccd1a6c31f9
I20260929 16:31:27.627950 31820 ts_tablet_manager.cc:1939] T 928d882b98294d298759dccd1a6c31f9 P abe72324831b4807b6c8e466cbc3b20f: Deleting consensus metadata
I20260929 16:31:27.628439 31823 tablet_replica.cc:330] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab: stopping tablet replica
I20260929 16:31:27.628568 31823 raft_consensus.cc:2229] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:27.628693 31823 raft_consensus.cc:2258] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:27.628741 31229 catalog_manager.cc:4841] TS abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253): tablet 928d882b98294d298759dccd1a6c31f9 (table test-table [id=0a988b9603a949cd88749ea12dac5b05]) successfully deleted
I20260929 16:31:27.628901 31821 tablet_replica.cc:330] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240: stopping tablet replica
I20260929 16:31:27.629027 31821 raft_consensus.cc:2229] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:27.629210 31821 raft_consensus.cc:2258] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:27.629839 31823 ts_tablet_manager.cc:1905] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:27.632567 31823 ts_tablet_manager.cc:1918] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:31:27.632647 31823 log.cc:1196] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/928d882b98294d298759dccd1a6c31f9
I20260929 16:31:27.632925 31823 ts_tablet_manager.cc:1939] T 928d882b98294d298759dccd1a6c31f9 P 34a905ebcbd04cb3b6ee0a7812602eab: Deleting consensus metadata
I20260929 16:31:27.633673 31229 catalog_manager.cc:4841] TS 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255): tablet 928d882b98294d298759dccd1a6c31f9 (table test-table [id=0a988b9603a949cd88749ea12dac5b05]) successfully deleted
I20260929 16:31:27.633895 31821 ts_tablet_manager.cc:1905] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:27.636911 31821 ts_tablet_manager.cc:1918] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:31:27.636991 31821 log.cc:1196] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/928d882b98294d298759dccd1a6c31f9
I20260929 16:31:27.637272 31821 ts_tablet_manager.cc:1939] T 928d882b98294d298759dccd1a6c31f9 P 3e7fa8e3288945eeb18b766e9a1a0240: Deleting consensus metadata
I20260929 16:31:27.638171 31230 catalog_manager.cc:4841] TS 3e7fa8e3288945eeb18b766e9a1a0240 (127.23.44.2:41643): tablet 928d882b98294d298759dccd1a6c31f9 (table test-table [id=0a988b9603a949cd88749ea12dac5b05]) successfully deleted
Sep 29 16:31:27 dist-test-slave-mlmz krb5kdc[31198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:31:27 dist-test-slave-mlmz krb5kdc[31198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:27 dist-test-slave-mlmz krb5kdc[31198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:31:27.664866 31292 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:27.657341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34750 (local address 127.23.44.62:40199)
0929 16:31:27.657496 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:27.657501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:27.657782 (+ 281us) server_negotiation.cc:407] Connection header received
0929 16:31:27.657956 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:27.657960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:27.658022 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:27.658138 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:27.658915 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.659832 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.660738 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.661079 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.661242 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.664740 (+ 3498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34750: BlockingRecv error: recv got EOF from 127.0.0.1:34750 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 16:31:27 dist-test-slave-mlmz krb5kdc[31198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM
I20260929 16:31:27.676380 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.666761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.23.44.62:40199)
0929 16:31:27.666878 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:27.666882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:27.667069 (+ 187us) server_negotiation.cc:407] Connection header received
0929 16:31:27.667185 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:27.667188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:27.667237 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:27.667327 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:27.668054 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.668823 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.669739 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.670074 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.670200 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.672940 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:27.672944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:27.672947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:27.672974 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:27.674671 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:27.675111 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:27.675115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:27.675117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:27.675205 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:27.675515 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:27.675520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:27.675525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:27.675745 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:27.675894 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:27.676104 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 16:31:27.676221 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:31:27.679006 31243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:27.679507 31243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:27.684847 31380 tablet_service.cc:1467] Processing CreateTablet for tablet 8602c7a0a8164a128b682c68cf783702 (DEFAULT_TABLE table=test-table [id=8d6f46fc62f94fa9a5fcfe6cb0b47402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:27.685070 31660 tablet_service.cc:1467] Processing CreateTablet for tablet 8602c7a0a8164a128b682c68cf783702 (DEFAULT_TABLE table=test-table [id=8d6f46fc62f94fa9a5fcfe6cb0b47402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:27.685169 31380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8602c7a0a8164a128b682c68cf783702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:27.685397 31660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8602c7a0a8164a128b682c68cf783702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:27.687430 31754 tablet_bootstrap.cc:492] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab: Bootstrap starting.
I20260929 16:31:27.687855 31520 tablet_service.cc:1467] Processing CreateTablet for tablet 8602c7a0a8164a128b682c68cf783702 (DEFAULT_TABLE table=test-table [id=8d6f46fc62f94fa9a5fcfe6cb0b47402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:27.688133 31520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8602c7a0a8164a128b682c68cf783702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:27.688431 31754 tablet_bootstrap.cc:654] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:27.689409 31755 tablet_bootstrap.cc:492] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f: Bootstrap starting.
I20260929 16:31:27.689631 31754 tablet_bootstrap.cc:492] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab: No bootstrap required, opened a new log
I20260929 16:31:27.689703 31754 ts_tablet_manager.cc:1397] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:27.690168 31754 raft_consensus.cc:348] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.690272 31755 tablet_bootstrap.cc:654] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:27.690296 31754 raft_consensus.cc:374] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:27.690328 31754 raft_consensus.cc:729] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a905ebcbd04cb3b6ee0a7812602eab, State: Initialized, Role: FOLLOWER
I20260929 16:31:27.690461 31754 consensus_queue.cc:260] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.690698 31756 tablet_bootstrap.cc:492] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240: Bootstrap starting.
I20260929 16:31:27.691408 31754 ts_tablet_manager.cc:1428] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 16:31:27.691519 31756 tablet_bootstrap.cc:654] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:27.692286 31755 tablet_bootstrap.cc:492] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f: No bootstrap required, opened a new log
I20260929 16:31:27.692387 31755 ts_tablet_manager.cc:1397] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 16:31:27.692868 31755 raft_consensus.cc:348] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.692986 31755 raft_consensus.cc:374] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:27.693022 31755 raft_consensus.cc:729] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe72324831b4807b6c8e466cbc3b20f, State: Initialized, Role: FOLLOWER
I20260929 16:31:27.693133 31755 consensus_queue.cc:260] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.693428 31755 ts_tablet_manager.cc:1428] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 16:31:27.693495 31756 tablet_bootstrap.cc:492] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240: No bootstrap required, opened a new log
I20260929 16:31:27.693578 31756 ts_tablet_manager.cc:1397] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 16:31:27.693957 31756 raft_consensus.cc:348] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.694058 31756 raft_consensus.cc:374] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:27.694088 31756 raft_consensus.cc:729] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7fa8e3288945eeb18b766e9a1a0240, State: Initialized, Role: FOLLOWER
I20260929 16:31:27.694205 31756 consensus_queue.cc:260] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.694446 31756 ts_tablet_manager.cc:1428] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 16:31:27.909586 31761 raft_consensus.cc:482] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:27.909812 31761 raft_consensus.cc:504] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.911060 31761 leader_election.cc:288] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e7fa8e3288945eeb18b766e9a1a0240 (127.23.44.2:41643), 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255)
I20260929 16:31:27.915508 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.911416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:51719 (local address 127.23.44.2:41643)
0929 16:31:27.911556 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:27.911559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:27.911766 (+ 207us) server_negotiation.cc:407] Connection header received
0929 16:31:27.911983 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:27.911989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:27.912050 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:27.912120 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:27.912492 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.913174 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.914410 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.914940 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.915018 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.915070 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:27.915351 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 16:31:27.915409 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:31:27.915622 31839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.911309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:41643 (local address 127.23.44.1:51719)
0929 16:31:27.911656 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:27.911672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:27.911835 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:27.912161 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:27.912167 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:27.912190 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:27.912381 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.912388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.913318 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.913321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.914270 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.914279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.915099 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.915104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.915207 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.915235 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:31:27.915354 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:31:27.915425 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":5,"thread_start_us":94,"threads_started":1}
I20260929 16:31:27.916109 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.911404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:48055 (local address 127.23.44.3:40255)
0929 16:31:27.911658 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:27.911665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:27.911917 (+ 252us) server_negotiation.cc:407] Connection header received
0929 16:31:27.912065 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:27.912067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:27.912140 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:27.912217 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:27.912937 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.913586 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.914804 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.915466 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.915571 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.915645 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:27.915903 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 16:31:27.915979 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260929 16:31:27.916154 31539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8602c7a0a8164a128b682c68cf783702" candidate_uuid: "abe72324831b4807b6c8e466cbc3b20f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" is_pre_election: true
I20260929 16:31:27.916308 31539 raft_consensus.cc:2454] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abe72324831b4807b6c8e466cbc3b20f in term 0.
I20260929 16:31:27.916471 31838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.911315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:40255 (local address 127.23.44.1:48055)
0929 16:31:27.911813 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:27.911827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:27.911962 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:27.912249 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:27.912252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:27.912268 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:27.912817 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.912827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.913729 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.913732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.914647 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:27.914657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.915598 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.915603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:27.915720 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.915741 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:31:27.916243 (+ 502us) client_negotiation.cc:240] Negotiation successful
0929 16:31:27.916314 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":82,"thread_start_us":97,"threads_started":1}
I20260929 16:31:27.916692 31334 leader_election.cc:302] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e7fa8e3288945eeb18b766e9a1a0240, abe72324831b4807b6c8e466cbc3b20f; no voters:
I20260929 16:31:27.916994 31680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8602c7a0a8164a128b682c68cf783702" candidate_uuid: "abe72324831b4807b6c8e466cbc3b20f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" is_pre_election: true
I20260929 16:31:27.917053 31761 raft_consensus.cc:2781] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:27.917104 31761 raft_consensus.cc:482] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:27.917145 31761 raft_consensus.cc:3037] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:27.917145 31680 raft_consensus.cc:2454] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abe72324831b4807b6c8e466cbc3b20f in term 0.
I20260929 16:31:27.918264 31761 raft_consensus.cc:504] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.918584 31761 leader_election.cc:288] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [CANDIDATE]: Term 1 election: Requested vote from peers 3e7fa8e3288945eeb18b766e9a1a0240 (127.23.44.2:41643), 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255)
I20260929 16:31:27.918886 31539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8602c7a0a8164a128b682c68cf783702" candidate_uuid: "abe72324831b4807b6c8e466cbc3b20f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e7fa8e3288945eeb18b766e9a1a0240"
I20260929 16:31:27.918917 31680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8602c7a0a8164a128b682c68cf783702" candidate_uuid: "abe72324831b4807b6c8e466cbc3b20f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a905ebcbd04cb3b6ee0a7812602eab"
I20260929 16:31:27.919009 31680 raft_consensus.cc:3037] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:27.919009 31539 raft_consensus.cc:3037] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:27.920092 31539 raft_consensus.cc:2454] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abe72324831b4807b6c8e466cbc3b20f in term 1.
I20260929 16:31:27.920092 31680 raft_consensus.cc:2454] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abe72324831b4807b6c8e466cbc3b20f in term 1.
I20260929 16:31:27.920367 31333 leader_election.cc:302] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a905ebcbd04cb3b6ee0a7812602eab, abe72324831b4807b6c8e466cbc3b20f; no voters:
I20260929 16:31:27.920578 31761 raft_consensus.cc:2781] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:27.920786 31761 raft_consensus.cc:686] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 1 LEADER]: Becoming Leader. State: Replica: abe72324831b4807b6c8e466cbc3b20f, State: Running, Role: LEADER
I20260929 16:31:27.921030 31761 consensus_queue.cc:237] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.923368 31242 catalog_manager.cc:5495] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f reported cstate change: term changed from 0 to 1, leader changed from <none> to abe72324831b4807b6c8e466cbc3b20f (127.23.44.1). New cstate: current_term: 1 leader_uuid: "abe72324831b4807b6c8e466cbc3b20f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } health_report { overall_health: UNKNOWN } } }
I20260929 16:31:27.932188 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:27.928709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.23.44.1:35253)
0929 16:31:27.928840 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:27.928844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:27.928856 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:31:27.929068 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:27.929072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:27.929150 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:27.929321 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:27.929553 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.930035 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.930831 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:27.931043 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:27.931132 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:27.931453 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:27.931898 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:27.931982 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:27.932060 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 16:31:27.932102 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:31:27.934873 31539 raft_consensus.cc:1261] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Refusing update from remote peer abe72324831b4807b6c8e466cbc3b20f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:27.934873 31680 raft_consensus.cc:1261] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Refusing update from remote peer abe72324831b4807b6c8e466cbc3b20f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:27.935372 31761 consensus_queue.cc:1035] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:27.935586 31840 consensus_queue.cc:1035] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:27.946017 31242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:27.946138 31242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:27.948057 31242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5: Sending DeleteTablet for 3 replicas of tablet 8602c7a0a8164a128b682c68cf783702
I20260929 16:31:27.948598 31380 tablet_service.cc:1514] Processing DeleteTablet for tablet 8602c7a0a8164a128b682c68cf783702 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:27 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:36116
I20260929 16:31:27.948801 31520 tablet_service.cc:1514] Processing DeleteTablet for tablet 8602c7a0a8164a128b682c68cf783702 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:27 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:54742
I20260929 16:31:27.948941 31660 tablet_service.cc:1514] Processing DeleteTablet for tablet 8602c7a0a8164a128b682c68cf783702 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:27 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260929 16:31:27.948973 31820 tablet_replica.cc:330] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f: stopping tablet replica
I20260929 16:31:27.949085 31820 raft_consensus.cc:2229] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:27.949110 31823 tablet_replica.cc:330] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab: stopping tablet replica
I20260929 16:31:27.949231 31823 raft_consensus.cc:2229] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:27.949344 31820 raft_consensus.cc:2258] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:27.949404 31823 raft_consensus.cc:2258] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:27.949513 31821 tablet_replica.cc:330] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240: stopping tablet replica
I20260929 16:31:27.949609 31821 raft_consensus.cc:2229] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:27.949801 31821 raft_consensus.cc:2258] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:27.950161 31823 ts_tablet_manager.cc:1905] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:27.950894 31821 ts_tablet_manager.cc:1905] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:27.952831 31823 ts_tablet_manager.cc:1918] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:27.952919 31823 log.cc:1196] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/8602c7a0a8164a128b682c68cf783702
I20260929 16:31:27.953147 31821 ts_tablet_manager.cc:1918] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:27.953214 31823 ts_tablet_manager.cc:1939] T 8602c7a0a8164a128b682c68cf783702 P 34a905ebcbd04cb3b6ee0a7812602eab: Deleting consensus metadata
I20260929 16:31:27.953238 31821 log.cc:1196] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/8602c7a0a8164a128b682c68cf783702
I20260929 16:31:27.953495 31821 ts_tablet_manager.cc:1939] T 8602c7a0a8164a128b682c68cf783702 P 3e7fa8e3288945eeb18b766e9a1a0240: Deleting consensus metadata
I20260929 16:31:27.953972 31229 catalog_manager.cc:4841] TS 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255): tablet 8602c7a0a8164a128b682c68cf783702 (table test-table [id=8d6f46fc62f94fa9a5fcfe6cb0b47402]) successfully deleted
I20260929 16:31:27.954077 31230 catalog_manager.cc:4841] TS 3e7fa8e3288945eeb18b766e9a1a0240 (127.23.44.2:41643): tablet 8602c7a0a8164a128b682c68cf783702 (table test-table [id=8d6f46fc62f94fa9a5fcfe6cb0b47402]) successfully deleted
I20260929 16:31:27.953972 31242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 16:31:27.954527 31820 ts_tablet_manager.cc:1905] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 16:31:27.954682 31242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:27.956660 31820 ts_tablet_manager.cc:1918] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:27.956746 31820 log.cc:1196] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/wal/wals/8602c7a0a8164a128b682c68cf783702
I20260929 16:31:27.957000 31820 ts_tablet_manager.cc:1939] T 8602c7a0a8164a128b682c68cf783702 P abe72324831b4807b6c8e466cbc3b20f: Deleting consensus metadata
I20260929 16:31:27.957669 31229 catalog_manager.cc:4841] TS abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253): tablet 8602c7a0a8164a128b682c68cf783702 (table test-table [id=8d6f46fc62f94fa9a5fcfe6cb0b47402]) successfully deleted
I20260929 16:31:27.959589 31380 tablet_service.cc:1467] Processing CreateTablet for tablet 88b254ea0cc0439db852ee3241087819 (DEFAULT_TABLE table=test-table [id=a8d98fb3f5e94d29a4f5bf9d8ebfc62d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:27.959827 31380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b254ea0cc0439db852ee3241087819. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:27.959842 31660 tablet_service.cc:1467] Processing CreateTablet for tablet 88b254ea0cc0439db852ee3241087819 (DEFAULT_TABLE table=test-table [id=a8d98fb3f5e94d29a4f5bf9d8ebfc62d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:27.960103 31660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b254ea0cc0439db852ee3241087819. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:27.961536 31755 tablet_bootstrap.cc:492] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f: Bootstrap starting.
I20260929 16:31:27.961706 31520 tablet_service.cc:1467] Processing CreateTablet for tablet 88b254ea0cc0439db852ee3241087819 (DEFAULT_TABLE table=test-table [id=a8d98fb3f5e94d29a4f5bf9d8ebfc62d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:27.961934 31520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b254ea0cc0439db852ee3241087819. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:27.962440 31755 tablet_bootstrap.cc:654] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:27.963989 31756 tablet_bootstrap.cc:492] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240: Bootstrap starting.
I20260929 16:31:27.964033 31754 tablet_bootstrap.cc:492] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab: Bootstrap starting.
I20260929 16:31:27.964759 31754 tablet_bootstrap.cc:654] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:27.964756 31756 tablet_bootstrap.cc:654] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:27.965886 31756 tablet_bootstrap.cc:492] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240: No bootstrap required, opened a new log
I20260929 16:31:27.965925 31754 tablet_bootstrap.cc:492] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab: No bootstrap required, opened a new log
I20260929 16:31:27.965988 31754 ts_tablet_manager.cc:1397] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 16:31:27.965970 31756 ts_tablet_manager.cc:1397] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:31:27.966359 31754 raft_consensus.cc:348] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.966377 31756 raft_consensus.cc:348] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.966465 31756 raft_consensus.cc:374] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:27.966465 31754 raft_consensus.cc:374] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:27.966498 31756 raft_consensus.cc:729] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7fa8e3288945eeb18b766e9a1a0240, State: Initialized, Role: FOLLOWER
I20260929 16:31:27.966498 31754 raft_consensus.cc:729] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a905ebcbd04cb3b6ee0a7812602eab, State: Initialized, Role: FOLLOWER
I20260929 16:31:27.966607 31756 consensus_queue.cc:260] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.966614 31754 consensus_queue.cc:260] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.966892 31754 ts_tablet_manager.cc:1428] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 16:31:27.966892 31756 ts_tablet_manager.cc:1428] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 16:31:27.967793 31755 tablet_bootstrap.cc:492] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f: No bootstrap required, opened a new log
I20260929 16:31:27.967875 31755 ts_tablet_manager.cc:1397] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260929 16:31:27.968315 31755 raft_consensus.cc:348] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.968432 31755 raft_consensus.cc:374] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:27.968461 31755 raft_consensus.cc:729] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe72324831b4807b6c8e466cbc3b20f, State: Initialized, Role: FOLLOWER
I20260929 16:31:27.968570 31755 consensus_queue.cc:260] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:27.968894 31755 ts_tablet_manager.cc:1428] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:31:28.017628 31762 raft_consensus.cc:482] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:28.018031 31762 raft_consensus.cc:504] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:28.018505 31762 leader_election.cc:288] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253), 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255)
I20260929 16:31:28.018934 31680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b254ea0cc0439db852ee3241087819" candidate_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" is_pre_election: true
I20260929 16:31:28.018961 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b254ea0cc0439db852ee3241087819" candidate_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe72324831b4807b6c8e466cbc3b20f" is_pre_election: true
I20260929 16:31:28.019093 31680 raft_consensus.cc:2454] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e7fa8e3288945eeb18b766e9a1a0240 in term 0.
I20260929 16:31:28.019127 31400 raft_consensus.cc:2454] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e7fa8e3288945eeb18b766e9a1a0240 in term 0.
I20260929 16:31:28.019429 31473 leader_election.cc:302] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e7fa8e3288945eeb18b766e9a1a0240, abe72324831b4807b6c8e466cbc3b20f; no voters:
I20260929 16:31:28.019601 31762 raft_consensus.cc:2781] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:28.019657 31762 raft_consensus.cc:482] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:28.019703 31762 raft_consensus.cc:3037] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:28.020701 31762 raft_consensus.cc:504] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:28.021054 31762 leader_election.cc:288] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [CANDIDATE]: Term 1 election: Requested vote from peers abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253), 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255)
I20260929 16:31:28.021397 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b254ea0cc0439db852ee3241087819" candidate_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe72324831b4807b6c8e466cbc3b20f"
I20260929 16:31:28.021418 31680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b254ea0cc0439db852ee3241087819" candidate_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a905ebcbd04cb3b6ee0a7812602eab"
I20260929 16:31:28.021508 31400 raft_consensus.cc:3037] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:28.021541 31680 raft_consensus.cc:3037] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:28.022567 31680 raft_consensus.cc:2454] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e7fa8e3288945eeb18b766e9a1a0240 in term 1.
I20260929 16:31:28.022588 31400 raft_consensus.cc:2454] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e7fa8e3288945eeb18b766e9a1a0240 in term 1.
I20260929 16:31:28.022821 31473 leader_election.cc:302] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e7fa8e3288945eeb18b766e9a1a0240, abe72324831b4807b6c8e466cbc3b20f; no voters:
I20260929 16:31:28.023023 31762 raft_consensus.cc:2781] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:28.023123 31762 raft_consensus.cc:686] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 LEADER]: Becoming Leader. State: Replica: 3e7fa8e3288945eeb18b766e9a1a0240, State: Running, Role: LEADER
I20260929 16:31:28.023263 31762 consensus_queue.cc:237] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } }
I20260929 16:31:28.024924 31241 catalog_manager.cc:5495] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e7fa8e3288945eeb18b766e9a1a0240 (127.23.44.2). New cstate: current_term: 1 leader_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e7fa8e3288945eeb18b766e9a1a0240" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 41643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 } health_report { overall_health: UNKNOWN } } }
I20260929 16:31:28.042196 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:28.038173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.23.44.2:41643)
0929 16:31:28.038557 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:28.038561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:28.038571 (+ 10us) server_negotiation.cc:407] Connection header received
0929 16:31:28.038603 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:28.038605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:28.038674 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:28.038742 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:28.039001 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:28.039507 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:28.040771 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:28.041057 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:28.041201 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:28.041634 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:28.041763 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:28.041897 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:28.042000 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 16:31:28.042060 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310}
I20260929 16:31:28.048039 31399 raft_consensus.cc:1261] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Refusing update from remote peer 3e7fa8e3288945eeb18b766e9a1a0240: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:28.048040 31680 raft_consensus.cc:1261] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Refusing update from remote peer 3e7fa8e3288945eeb18b766e9a1a0240: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:28.048512 31773 consensus_queue.cc:1035] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe72324831b4807b6c8e466cbc3b20f" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:28.048686 31762 consensus_queue.cc:1035] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a905ebcbd04cb3b6ee0a7812602eab" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:28.082973 31241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:28.083184 31241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:28.085510 31241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f75b20fa5de4a268c1efd66287ca0b5: Sending DeleteTablet for 3 replicas of tablet 88b254ea0cc0439db852ee3241087819
I20260929 16:31:28.086059 31520 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b254ea0cc0439db852ee3241087819 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:28 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:54742
I20260929 16:31:28.086089 31380 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b254ea0cc0439db852ee3241087819 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:28 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:36116
I20260929 16:31:28.086246 31821 tablet_replica.cc:330] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240: stopping tablet replica
I20260929 16:31:28.086277 31820 tablet_replica.cc:330] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f: stopping tablet replica
I20260929 16:31:28.086361 31821 raft_consensus.cc:2229] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:28.086396 31820 raft_consensus.cc:2229] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:28.086421 31660 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b254ea0cc0439db852ee3241087819 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:28 UTC) from {username='oryx', principal='oryx/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260929 16:31:28.086503 31820 raft_consensus.cc:2258] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:28.086553 31821 raft_consensus.cc:2258] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:28.086680 31823 tablet_replica.cc:330] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab: stopping tablet replica
I20260929 16:31:28.086788 31823 raft_consensus.cc:2229] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:28.086894 31823 raft_consensus.cc:2258] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:28.087532 31820 ts_tablet_manager.cc:1905] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:28.087559 31823 ts_tablet_manager.cc:1905] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:28.090147 31820 ts_tablet_manager.cc:1918] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:31:28.090148 31823 ts_tablet_manager.cc:1918] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:31:28.090168 31821 ts_tablet_manager.cc:1905] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:28.090226 31823 log.cc:1196] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/88b254ea0cc0439db852ee3241087819
I20260929 16:31:28.090240 31820 log.cc:1196] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-0/wal/wals/88b254ea0cc0439db852ee3241087819
I20260929 16:31:28.090505 31823 ts_tablet_manager.cc:1939] T 88b254ea0cc0439db852ee3241087819 P 34a905ebcbd04cb3b6ee0a7812602eab: Deleting consensus metadata
I20260929 16:31:28.090512 31820 ts_tablet_manager.cc:1939] T 88b254ea0cc0439db852ee3241087819 P abe72324831b4807b6c8e466cbc3b20f: Deleting consensus metadata
I20260929 16:31:28.091416 31229 catalog_manager.cc:4841] TS abe72324831b4807b6c8e466cbc3b20f (127.23.44.1:35253): tablet 88b254ea0cc0439db852ee3241087819 (table test-table [id=a8d98fb3f5e94d29a4f5bf9d8ebfc62d]) successfully deleted
I20260929 16:31:28.091583 31229 catalog_manager.cc:4841] TS 34a905ebcbd04cb3b6ee0a7812602eab (127.23.44.3:40255): tablet 88b254ea0cc0439db852ee3241087819 (table test-table [id=a8d98fb3f5e94d29a4f5bf9d8ebfc62d]) successfully deleted
I20260929 16:31:28.092202 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 31315
I20260929 16:31:28.092832 31821 ts_tablet_manager.cc:1918] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:31:28.092916 31821 log.cc:1196] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/88b254ea0cc0439db852ee3241087819
I20260929 16:31:28.093129 31821 ts_tablet_manager.cc:1939] T 88b254ea0cc0439db852ee3241087819 P 3e7fa8e3288945eeb18b766e9a1a0240: Deleting consensus metadata
I20260929 16:31:28.093801 31230 catalog_manager.cc:4841] TS 3e7fa8e3288945eeb18b766e9a1a0240 (127.23.44.2:41643): tablet 88b254ea0cc0439db852ee3241087819 (table test-table [id=a8d98fb3f5e94d29a4f5bf9d8ebfc62d]) successfully deleted
I20260929 16:31:28.100244 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 31455
I20260929 16:31:28.107339 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 31595
I20260929 16:31:28.114478 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 31211
2026-09-29T16:31:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3419 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:28 dist-test-slave-mlmz krb5kdc[31856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:28 dist-test-slave-mlmz krb5kdc[31856](info): set up 2 sockets
Sep 29 16:31:28 dist-test-slave-mlmz krb5kdc[31856](info): commencing operation
krb5kdc: starting...
W20260929 16:31:29.145985 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:31:29 dist-test-slave-mlmz krb5kdc[31856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:31:29.264504 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:42543
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:42543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:29.372854 31869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:29.373112 31869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:29.373152 31869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:29.377120 31869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:29.377182 31869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:29.377200 31869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:29.377216 31869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:29.377265 31869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:29.382522 31869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:42543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:42543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:29.383558 31869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:29.384549 31869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:29.394887 31875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:29.394907 31874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:29.395010 31869 server_base.cc:1034] running on GCE node
W20260929 16:31:29.394872 31877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:29.395643 31869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:29.396731 31869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:29.397972 31869 hybrid_clock.cc:648] HybridClock initialized: now 1790699489397965 us; error 46 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T16:31:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1296 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:29 dist-test-slave-mlmz krb5kdc[31882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:29 dist-test-slave-mlmz krb5kdc[31882](info): set up 2 sockets
Sep 29 16:31:29 dist-test-slave-mlmz krb5kdc[31882](info): commencing operation
krb5kdc: starting...
W20260929 16:31:30.449918 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:31:30 dist-test-slave-mlmz krb5kdc[31882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:31:30.573236 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:42249
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:42249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:30.681437 31895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:30.681684 31895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:30.681725 31895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:30.685714 31895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:30.685765 31895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:30.685787 31895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:30.685803 31895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:30.685858 31895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:30.690984 31895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:42249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:42249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:30.691980 31895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:30.692946 31895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:30.701411 31900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:30.701624 31903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:30.701715 31901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:30.701798 31895 server_base.cc:1034] running on GCE node
I20260929 16:31:30.702284 31895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:30.703289 31895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:30.704494 31895 hybrid_clock.cc:648] HybridClock initialized: now 1790699490704477 us; error 29 us; skew 500 ppm
Sep 29 16:31:30 dist-test-slave-mlmz krb5kdc[31882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699490, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:30.707217 31895 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:31:30.707994 31895 webserver.cc:460] Webserver started at http://127.23.44.62:41503/ using document root <none> and password file <none>
I20260929 16:31:30.708546 31895 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:30.708606 31895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:30.708768 31895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:30.710718 31895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "7a5461b2cf4f4bf5962fa93ec80d155f"
format_stamp: "Formatted at 2026-09-29 16:31:30 on dist-test-slave-mlmz"
server_key: "029ca47a9196caae60c90710011830d3"
server_key_iv: "436d11c668804b421ff80d724f515e0b"
server_key_version: "encryptionkey@0"
I20260929 16:31:30.711213 31895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "7a5461b2cf4f4bf5962fa93ec80d155f"
format_stamp: "Formatted at 2026-09-29 16:31:30 on dist-test-slave-mlmz"
server_key: "029ca47a9196caae60c90710011830d3"
server_key_iv: "436d11c668804b421ff80d724f515e0b"
server_key_version: "encryptionkey@0"
I20260929 16:31:30.714753 31895 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:31:30.717341 31910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:30.718653 31895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:31:30.718775 31895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "7a5461b2cf4f4bf5962fa93ec80d155f"
format_stamp: "Formatted at 2026-09-29 16:31:30 on dist-test-slave-mlmz"
server_key: "029ca47a9196caae60c90710011830d3"
server_key_iv: "436d11c668804b421ff80d724f515e0b"
server_key_version: "encryptionkey@0"
I20260929 16:31:30.718863 31895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:30.730283 31895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:30.733462 31895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:30.733604 31895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:30.740309 31895 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:42249
I20260929 16:31:30.740336 31962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:42249 every 8 connection(s)
I20260929 16:31:30.741199 31895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:30.743919 31963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:30.749078 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 31895
I20260929 16:31:30.749189 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:30.749409 31963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f: Bootstrap starting.
I20260929 16:31:30.749532 23728 external_mini_cluster.cc:1428] Setting key 28b68e50bbbce0844ae32d3a2b321af9
I20260929 16:31:30.751865 31963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:30.752588 31963 log.cc:824] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:30.754398 31963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f: No bootstrap required, opened a new log
Sep 29 16:31:30 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:30.758005 31963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42249 } }
I20260929 16:31:30.758244 31963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:30.758306 31963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5461b2cf4f4bf5962fa93ec80d155f, State: Initialized, Role: FOLLOWER
I20260929 16:31:30.758791 31963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42249 } }
I20260929 16:31:30.758944 31963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:30.759030 31963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:30.759151 31963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:30.760242 31963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42249 } }
I20260929 16:31:30.760594 31963 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5461b2cf4f4bf5962fa93ec80d155f; no voters:
I20260929 16:31:30.760950 31963 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:30.761222 31968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:30.761689 31968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 1 LEADER]: Becoming Leader. State: Replica: 7a5461b2cf4f4bf5962fa93ec80d155f, State: Running, Role: LEADER
I20260929 16:31:30.762151 31968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42249 } }
I20260929 16:31:30.762812 31963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:30.763413 31968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5461b2cf4f4bf5962fa93ec80d155f. Latest consensus state: current_term: 1 leader_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42249 } } }
I20260929 16:31:30.763628 31968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:30.763953 31968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42249 } } }
I20260929 16:31:30.764111 31968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:30.764359 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:30.750881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.23.44.62:42249)
0929 16:31:30.751298 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:30.751312 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:30.751346 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:31:30.752023 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:30.752054 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:30.752373 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:30.752696 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:30.753531 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:30.754532 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:30.755396 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:30.755873 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:30.756046 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:30.758800 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:30.758805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:30.758818 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:30.758844 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:30.760864 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:30.761416 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:30.761422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:30.761428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:30.761553 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:30.761857 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:30.761860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:30.761862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:30.762247 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:30.762445 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:30.763070 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 16:31:30.763366 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":132,"threads_started":1}
W20260929 16:31:30.768054 31984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:31:30.768141 31984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:31:30.768220 31975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:30.769183 31975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:30.775195 31975 catalog_manager.cc:1347] Generated new cluster ID: 90b12a3f6c7b4826b0efb09435f3d90e
I20260929 16:31:30.775272 31975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:30.783345 31975 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:30.784224 31975 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:30.788187 31975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f: Generated new TSK 0
I20260929 16:31:30.788825 31975 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:31:30.823940 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:42249
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:30.931466 31989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:30.931699 31989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:30.931740 31989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:30.935802 31989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:30.935858 31989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:30.935936 31989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:30.941077 31989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:30.942054 31989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:30.942991 31989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:30.952481 31994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:30.952477 31995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:30.952615 31997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:30.952775 31989 server_base.cc:1034] running on GCE node
I20260929 16:31:30.953383 31989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:30.953997 31989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:30.955201 31989 hybrid_clock.cc:648] HybridClock initialized: now 1790699490955187 us; error 27 us; skew 500 ppm
Sep 29 16:31:30 dist-test-slave-mlmz krb5kdc[31882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699490, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:30.957960 31989 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:31:30.958895 31989 webserver.cc:460] Webserver started at http://127.23.44.1:35077/ using document root <none> and password file <none>
I20260929 16:31:30.959466 31989 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:30.959554 31989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:30.959795 31989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:30.961870 31989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "68f0e9171bb64a7cb6638b3fd2cea179"
format_stamp: "Formatted at 2026-09-29 16:31:30 on dist-test-slave-mlmz"
server_key: "3d74d2087da2199cd357b071b3bc1581"
server_key_iv: "2a4969a644b254275606b85a2b7cdad4"
server_key_version: "encryptionkey@0"
I20260929 16:31:30.962427 31989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "68f0e9171bb64a7cb6638b3fd2cea179"
format_stamp: "Formatted at 2026-09-29 16:31:30 on dist-test-slave-mlmz"
server_key: "3d74d2087da2199cd357b071b3bc1581"
server_key_iv: "2a4969a644b254275606b85a2b7cdad4"
server_key_version: "encryptionkey@0"
I20260929 16:31:30.966260 31989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:31:30.968662 32004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:30.969974 31989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:30.970114 31989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "68f0e9171bb64a7cb6638b3fd2cea179"
format_stamp: "Formatted at 2026-09-29 16:31:30 on dist-test-slave-mlmz"
server_key: "3d74d2087da2199cd357b071b3bc1581"
server_key_iv: "2a4969a644b254275606b85a2b7cdad4"
server_key_version: "encryptionkey@0"
I20260929 16:31:30.970224 31989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:30.983013 31989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:30.986159 31989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:30.986366 31989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:30.986963 31989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:30.987777 31989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:30.987848 31989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:30.987919 31989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:30.987963 31989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:30.997689 32117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:38635 every 8 connection(s)
I20260929 16:31:30.997718 31989 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:38635
I20260929 16:31:30.998670 31989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:30.999931 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 31989
I20260929 16:31:31.000094 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:31.000427 23728 external_mini_cluster.cc:1428] Setting key 175ef822578833b6f97d9a5b99963fab
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699490, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
I20260929 16:31:31.013423 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:31.000584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:40825 (local address 127.23.44.62:42249)
0929 16:31:31.000726 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:31.000738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:31.002338 (+ 1600us) server_negotiation.cc:407] Connection header received
0929 16:31:31.003162 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:31.003168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:31.003231 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:31.003332 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:31.004950 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.005560 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.006775 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.007017 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.007128 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:31.009628 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:31.009631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:31.009634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:31.009665 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:31.011260 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:31.011769 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:31.011774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:31.011776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:31.011893 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:31.012232 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:31.012236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:31.012237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:31.012436 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:31.012539 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:31.013120 (+ 581us) server_negotiation.cc:299] Negotiation successful
0929 16:31:31.013225 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:31:31.014516 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:31.001002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:40825)
0929 16:31:31.002188 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:31.002238 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:31.002965 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:31.003541 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:31.003552 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:31.004073 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:31.004760 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:31.004801 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.005681 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.005685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:31.006647 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:31.006657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.007163 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.007168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:31.007338 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:31.007944 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:31.007960 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:31.009464 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:31.011385 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:31.011391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:31.011405 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:31.011656 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:31.011985 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:31.011989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:31.011991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:31.012129 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:31.012538 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:31.012545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:31.012851 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 16:31:31.013070 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 16:31:31.013467 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":190,"threads_started":1}
I20260929 16:31:31.015801 32118 heartbeater.cc:347] Connected to a master server at 127.23.44.62:42249
I20260929 16:31:31.016052 32118 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:31.016536 32118 heartbeater.cc:511] Master 127.23.44.62:42249 requested a full tablet report, sending...
I20260929 16:31:31.018217 31927 ts_manager.cc:194] Registered new tserver with Master: 68f0e9171bb64a7cb6638b3fd2cea179 (127.23.44.1:38635)
I20260929 16:31:31.019801 31927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:40825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:31:31.036564 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:42249
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:31.164284 32123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:31.164517 32123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:31.164558 32123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:31.168576 32123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:31.168632 32123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:31.168690 32123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:31.173862 32123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:31.174840 32123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:31.175752 32123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:31:31.185675 32123 server_base.cc:1034] running on GCE node
W20260929 16:31:31.185631 32131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:31.185622 32128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:31.185622 32129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:31.186291 32123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:31.186925 32123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:31.188123 32123 hybrid_clock.cc:648] HybridClock initialized: now 1790699491188101 us; error 34 us; skew 500 ppm
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699491, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:31.190944 32123 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:31:31.191737 32123 webserver.cc:460] Webserver started at http://127.23.44.2:43339/ using document root <none> and password file <none>
I20260929 16:31:31.192282 32123 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:31.192373 32123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:31.192595 32123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:31.194590 32123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "598b2e758c51448d805f06473a6b4aea"
format_stamp: "Formatted at 2026-09-29 16:31:31 on dist-test-slave-mlmz"
server_key: "e45d14f23ad9af65c81601a1c6b08898"
server_key_iv: "f6998ae88e85f7a0a2536bfdc528cd48"
server_key_version: "encryptionkey@0"
I20260929 16:31:31.195138 32123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "598b2e758c51448d805f06473a6b4aea"
format_stamp: "Formatted at 2026-09-29 16:31:31 on dist-test-slave-mlmz"
server_key: "e45d14f23ad9af65c81601a1c6b08898"
server_key_iv: "f6998ae88e85f7a0a2536bfdc528cd48"
server_key_version: "encryptionkey@0"
I20260929 16:31:31.198748 32123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:31:31.201143 32138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:31.202687 32123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:31:31.202828 32123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "598b2e758c51448d805f06473a6b4aea"
format_stamp: "Formatted at 2026-09-29 16:31:31 on dist-test-slave-mlmz"
server_key: "e45d14f23ad9af65c81601a1c6b08898"
server_key_iv: "f6998ae88e85f7a0a2536bfdc528cd48"
server_key_version: "encryptionkey@0"
I20260929 16:31:31.202960 32123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:31.213382 32123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:31.216467 32123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:31.216653 32123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:31.217192 32123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:31.218015 32123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:31.218084 32123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:31.218150 32123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:31.218204 32123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:31.227489 32123 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:37985
I20260929 16:31:31.227520 32251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:37985 every 8 connection(s)
I20260929 16:31:31.228312 32123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:31.236673 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 32123
I20260929 16:31:31.236833 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:31.237113 23728 external_mini_cluster.cc:1428] Setting key ce773ed810f3854fe23c2b8bec9aa2b2
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699491, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:31.241837 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:31.230219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:34131 (local address 127.23.44.62:42249)
0929 16:31:31.230468 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:31.230471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:31.231288 (+ 817us) server_negotiation.cc:407] Connection header received
0929 16:31:31.232196 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:31.232200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:31.232253 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:31.232334 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:31.233554 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.234142 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.235253 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.235480 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.235601 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:31.238065 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:31.238069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:31.238071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:31.238097 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:31.239506 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:31.240126 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:31.240131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:31.240134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:31.240230 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:31.240543 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:31.240547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:31.240549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:31.240780 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:31.240893 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:31.241424 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 16:31:31.241561 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260929 16:31:31.242439 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:31.230487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:34131)
0929 16:31:31.231137 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:31.231175 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:31.231960 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:31.232494 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:31.232500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:31.232893 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:31.233418 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:31.233432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.234271 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.234274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:31.235128 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:31.235137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.235626 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.235631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:31.235799 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:31.236301 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:31.236316 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:31.237912 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:31.239620 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:31.239628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:31.239645 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:31.240006 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:31.240323 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:31.240325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:31.240326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:31.240451 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:31.240903 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:31.240908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:31.241143 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 16:31:31.241362 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 16:31:31.241620 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":114,"threads_started":1}
I20260929 16:31:31.243556 32252 heartbeater.cc:347] Connected to a master server at 127.23.44.62:42249
I20260929 16:31:31.243804 32252 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:31.244298 32252 heartbeater.cc:511] Master 127.23.44.62:42249 requested a full tablet report, sending...
I20260929 16:31:31.245307 31927 ts_manager.cc:194] Registered new tserver with Master: 598b2e758c51448d805f06473a6b4aea (127.23.44.2:37985)
I20260929 16:31:31.245878 31927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:34131
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:31:31.264068 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:42249
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:31.371536 32257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:31.371809 32257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:31.371881 32257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:31.375926 32257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:31.375996 32257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:31.376080 32257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:31.381074 32257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:31.382046 32257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:31.382970 32257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:31.391887 32265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:31.391888 32262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:31.392030 32263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:31.392110 32257 server_base.cc:1034] running on GCE node
I20260929 16:31:31.392505 32257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:31.393796 32257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:31.395028 32257 hybrid_clock.cc:648] HybridClock initialized: now 1790699491395015 us; error 34 us; skew 500 ppm
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699491, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:31.397763 32257 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:31:31.398653 32257 webserver.cc:460] Webserver started at http://127.23.44.3:34489/ using document root <none> and password file <none>
I20260929 16:31:31.399235 32257 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:31.399346 32257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:31.399571 32257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:31.401602 32257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "235d8f511d2e4c1d804c41e517e117b0"
format_stamp: "Formatted at 2026-09-29 16:31:31 on dist-test-slave-mlmz"
server_key: "85e40f97dc13080c97ccb37b71cf569a"
server_key_iv: "d1741b004bc61e011b447092738f9125"
server_key_version: "encryptionkey@0"
I20260929 16:31:31.402148 32257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "235d8f511d2e4c1d804c41e517e117b0"
format_stamp: "Formatted at 2026-09-29 16:31:31 on dist-test-slave-mlmz"
server_key: "85e40f97dc13080c97ccb37b71cf569a"
server_key_iv: "d1741b004bc61e011b447092738f9125"
server_key_version: "encryptionkey@0"
I20260929 16:31:31.405773 32257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:31:31.408176 32272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:31.409451 32257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:31.409572 32257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "235d8f511d2e4c1d804c41e517e117b0"
format_stamp: "Formatted at 2026-09-29 16:31:31 on dist-test-slave-mlmz"
server_key: "85e40f97dc13080c97ccb37b71cf569a"
server_key_iv: "d1741b004bc61e011b447092738f9125"
server_key_version: "encryptionkey@0"
I20260929 16:31:31.409675 32257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:31.422683 32257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:31.425477 32257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:31.425666 32257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:31.426189 32257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:31.426950 32257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:31.427021 32257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:31.427086 32257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:31.427134 32257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:31.436390 32257 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:37259
I20260929 16:31:31.436407 32385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:37259 every 8 connection(s)
I20260929 16:31:31.437403 32257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:31.439973 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 32257
I20260929 16:31:31.440105 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:31.440435 23728 external_mini_cluster.cc:1428] Setting key afce25bdf6392226bde699515be57cb0
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699491, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:31.450177 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:31.439023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:48463 (local address 127.23.44.62:42249)
0929 16:31:31.439156 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:31.439160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:31.440223 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 16:31:31.441400 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:31.441404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:31.441462 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:31.441531 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:31.442766 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.443285 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.444370 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.444651 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.444731 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:31.446990 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:31.446993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:31.446996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:31.447018 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:31.448404 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:31.448834 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:31.448837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:31.448838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:31.448909 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:31.449206 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:31.449209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:31.449210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:31.449465 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:31.449561 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:31.449977 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 16:31:31.450066 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:31:31.450857 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:31.439376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:48463)
0929 16:31:31.440072 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:31.440127 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:31.441137 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:31.441707 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:31.441716 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:31.442085 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:31.442647 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:31.442661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.443396 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.443398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:31.444215 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:31.444223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.444770 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.444772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:31.444924 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:31.445476 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:31.445491 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:31.446853 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:31.448498 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:31.448502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:31.448511 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:31.448732 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:31.448985 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:31.448987 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:31.448989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:31.449104 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:31.449566 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:31.449571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:31.449759 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 16:31:31.449923 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 16:31:31.450134 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":103,"threads_started":1}
I20260929 16:31:31.451922 32386 heartbeater.cc:347] Connected to a master server at 127.23.44.62:42249
I20260929 16:31:31.452131 32386 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:31.452572 32386 heartbeater.cc:511] Master 127.23.44.62:42249 requested a full tablet report, sending...
I20260929 16:31:31.453552 31927 ts_manager.cc:194] Registered new tserver with Master: 235d8f511d2e4c1d804c41e517e117b0 (127.23.44.3:37259)
I20260929 16:31:31.454210 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:31:31.454268 31927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:48463
WARNING: no policy specified for oryx/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab.
Entry for principal oryx/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab.
WARNING: no policy specified for oryx/127.23.44.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab.
Entry for principal oryx/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab.
WARNING: no policy specified for oryx/127.23.44.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab.
Entry for principal oryx/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab.
I20260929 16:31:31.524529 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 31989
I20260929 16:31:31.529911 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 32123
I20260929 16:31:31.535162 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 32257
I20260929 16:31:31.540257 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 31895
I20260929 16:31:31.546667 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:42249
--webserver_interface=127.23.44.62
--webserver_port=41503
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:42249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:31.653833 32395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:31.654074 32395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:31.654114 32395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:31.658147 32395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:31.658196 32395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:31.658216 32395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:31.658232 32395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:31.658270 32395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:31.663444 32395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:42249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:42249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=41503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:31.664407 32395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:31.665431 32395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:31.673548 32400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:31.673502 32401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:31.673496 32403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:31.673718 32395 server_base.cc:1034] running on GCE node
I20260929 16:31:31.674427 32395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:31.675519 32395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:31.676703 32395 hybrid_clock.cc:648] HybridClock initialized: now 1790699491676668 us; error 51 us; skew 500 ppm
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699491, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:31.679328 32395 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:31:31.680218 32395 webserver.cc:460] Webserver started at http://127.23.44.62:41503/ using document root <none> and password file <none>
I20260929 16:31:31.680763 32395 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:31.680866 32395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:31.684536 32395 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:31.686657 32410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:31.687896 32395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:31:31.688030 32395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "7a5461b2cf4f4bf5962fa93ec80d155f"
format_stamp: "Formatted at 2026-09-29 16:31:30 on dist-test-slave-mlmz"
server_key: "029ca47a9196caae60c90710011830d3"
server_key_iv: "436d11c668804b421ff80d724f515e0b"
server_key_version: "encryptionkey@0"
I20260929 16:31:31.688555 32395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:31.707620 32395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:31.710611 32395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:31.710848 32395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:31.718190 32395 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:42249
I20260929 16:31:31.718214 32462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:42249 every 8 connection(s)
I20260929 16:31:31.719213 32395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:31.722369 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 32395
I20260929 16:31:31.722818 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:38635
--local_ip_for_outbound_sockets=127.23.44.1
--tserver_master_addrs=127.23.44.62:42249
--webserver_port=35077
--webserver_interface=127.23.44.1
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.44.1
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:31:31.723516 32463 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:31:31.725077 32463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f: Bootstrap starting.
I20260929 16:31:31.744160 32463 log.cc:824] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:31.752020 32463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:31:31.752453 32463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f: Bootstrap complete.
I20260929 16:31:31.757838 32463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42249 } }
I20260929 16:31:31.758329 32463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5461b2cf4f4bf5962fa93ec80d155f, State: Initialized, Role: FOLLOWER
I20260929 16:31:31.758884 32463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42249 } }
I20260929 16:31:31.759063 32463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:31.759147 32463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:31.759250 32463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:31:31.761813 32463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42249 } }
I20260929 16:31:31.762188 32463 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5461b2cf4f4bf5962fa93ec80d155f; no voters:
I20260929 16:31:31.762506 32463 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:31:31.762884 32468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:31:31.763151 32468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [term 2 LEADER]: Becoming Leader. State: Replica: 7a5461b2cf4f4bf5962fa93ec80d155f, State: Running, Role: LEADER
I20260929 16:31:31.763499 32468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42249 } }
I20260929 16:31:31.763716 32463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:31.764392 32468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5461b2cf4f4bf5962fa93ec80d155f. Latest consensus state: current_term: 2 leader_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42249 } } }
I20260929 16:31:31.764511 32468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:31.764916 32467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5461b2cf4f4bf5962fa93ec80d155f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42249 } } }
I20260929 16:31:31.765021 32467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:31.765477 32475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:31.768534 32475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:31.769200 32475 catalog_manager.cc:1259] Loaded cluster ID: 90b12a3f6c7b4826b0efb09435f3d90e
I20260929 16:31:31.769338 32475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:31.771692 32475 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:31.772368 32475 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7a5461b2cf4f4bf5962fa93ec80d155f: Loaded TSK: 0
I20260929 16:31:31.773629 32475 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:31:31.841239 32465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:31.841536 32465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:31.841578 32465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:31.845696 32465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:31.845752 32465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:31.845839 32465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:31.851099 32465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.1.keytab
--principal=oryx/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:38635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=35077
--tserver_master_addrs=127.23.44.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:31.852053 32465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:31.852996 32465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:31.863214 32490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:31.863231 32489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:31.863221 32492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:31.863524 32465 server_base.cc:1034] running on GCE node
I20260929 16:31:31.864027 32465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:31.864854 32465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:31.866044 32465 hybrid_clock.cc:648] HybridClock initialized: now 1790699491866019 us; error 44 us; skew 500 ppm
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699491, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:31.868782 32465 init.cc:377] Logged in from keytab as oryx/127.23.44.1@KRBTEST.COM (short username oryx)
I20260929 16:31:31.869787 32465 webserver.cc:460] Webserver started at http://127.23.44.1:35077/ using document root <none> and password file <none>
I20260929 16:31:31.870360 32465 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:31.870431 32465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:31.874146 32465 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:31.876250 32499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:31.877542 32465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:31.877678 32465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "68f0e9171bb64a7cb6638b3fd2cea179"
format_stamp: "Formatted at 2026-09-29 16:31:30 on dist-test-slave-mlmz"
server_key: "3d74d2087da2199cd357b071b3bc1581"
server_key_iv: "2a4969a644b254275606b85a2b7cdad4"
server_key_version: "encryptionkey@0"
I20260929 16:31:31.878203 32465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:31.908927 32465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:31.912178 32465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:31.912386 32465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:31.912976 32465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:31.913805 32465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:31.913875 32465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:31.913945 32465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:31.913991 32465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:31.923013 32465 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:38635
I20260929 16:31:31.923044 32612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:38635 every 8 connection(s)
I20260929 16:31:31.923938 32465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:31.929344 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 32465
I20260929 16:31:31.929776 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:37985
--local_ip_for_outbound_sockets=127.23.44.2
--tserver_master_addrs=127.23.44.62:42249
--webserver_port=43339
--webserver_interface=127.23.44.2
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.44.2
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:31:31.938161 32615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:31.925938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:39249 (local address 127.23.44.62:42249)
0929 16:31:31.926517 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:31.926537 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:31.926959 (+ 422us) server_negotiation.cc:407] Connection header received
0929 16:31:31.928071 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:31.928081 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:31.928370 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:31.928701 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:31.930033 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.931104 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.932758 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.933582 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.933787 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:31.937443 (+ 3656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:39249: BlockingRecv error: recv got EOF from 127.23.44.1:39249 (error 108)
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":133,"threads_started":1}
I20260929 16:31:31.938242 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:31.926038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:39249)
0929 16:31:31.926772 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:31.926835 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:31.927659 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:31.928788 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:31.928796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:31.929233 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:31.929800 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:31.929814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.931251 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.931257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:31.932582 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:31.932592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.933764 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.933770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:31.933948 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:31.934571 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:31.934586 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:31.937186 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":112,"threads_started":1}
W20260929 16:31:31.938843 32613 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:42249 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.44.62:42249: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:31.944734 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:31.939310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:39621)
0929 16:31:31.939501 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:31.939516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:31.939611 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:31.939859 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:31.939865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:31.940052 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:31.940188 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:31.940197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.940876 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.940881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:31.941866 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:31.941877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.942281 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.942284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:31.942374 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:31.942792 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:31.942806 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:31.944609 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107}
W20260929 16:31:31.945086 32615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:31.939390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:39621 (local address 127.23.44.62:42249)
0929 16:31:31.939499 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:31.939502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:31.939563 (+ 61us) server_negotiation.cc:407] Connection header received
0929 16:31:31.939712 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:31.939714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:31.939753 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:31.939813 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:31.940301 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.940769 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.941976 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.942179 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.942269 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:31.944763 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:39621: BlockingRecv error: recv got EOF from 127.23.44.1:39621 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:31 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:31.950739 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:31.945188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:41451)
0929 16:31:31.945411 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:31.945423 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:31.945487 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:31.945834 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:31.945837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:31.945986 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:31.946063 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:31.946069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.946787 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.946790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:31.947659 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:31.947669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.948128 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.948133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:31.948239 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:31.948688 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:31.948701 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:31.950593 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163}
W20260929 16:31:31.950904 32613 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:31:31.950999 32615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:31.945517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:41451 (local address 127.23.44.62:42249)
0929 16:31:31.945642 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:31.945646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:31.945658 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:31:31.945689 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:31.945691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:31.945725 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:31.945807 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:31.946166 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.946688 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.947807 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:31.948008 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:31.948093 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:31.950721 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:41451: BlockingRecv error: recv got EOF from 127.23.44.1:41451 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 16:31:32.040982 32617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:32.041261 32617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:32.041356 32617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:32.045285 32617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:32.045372 32617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:32.045461 32617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:32.050544 32617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.2.keytab
--principal=oryx/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:37985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=43339
--tserver_master_addrs=127.23.44.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:32.051514 32617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:32.052474 32617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:32.061671 32622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:32.061718 32625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:32.061870 32617 server_base.cc:1034] running on GCE node
W20260929 16:31:32.061761 32623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:32.062443 32617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:32.063059 32617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:32.064268 32617 hybrid_clock.cc:648] HybridClock initialized: now 1790699492064237 us; error 44 us; skew 500 ppm
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699492, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:32.067219 32617 init.cc:377] Logged in from keytab as oryx/127.23.44.2@KRBTEST.COM (short username oryx)
I20260929 16:31:32.068181 32617 webserver.cc:460] Webserver started at http://127.23.44.2:43339/ using document root <none> and password file <none>
I20260929 16:31:32.068768 32617 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:32.068889 32617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:32.073016 32617 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:31:32.075243 32632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:32.076565 32617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:32.076740 32617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "598b2e758c51448d805f06473a6b4aea"
format_stamp: "Formatted at 2026-09-29 16:31:31 on dist-test-slave-mlmz"
server_key: "e45d14f23ad9af65c81601a1c6b08898"
server_key_iv: "f6998ae88e85f7a0a2536bfdc528cd48"
server_key_version: "encryptionkey@0"
I20260929 16:31:32.077389 32617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:32.095825 32617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:32.099071 32617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:32.099287 32617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:32.099911 32617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:32.100795 32617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:32.100870 32617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:32.100947 32617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:32.100996 32617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:32.110536 32617 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:37985
I20260929 16:31:32.110565 32745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:37985 every 8 connection(s)
I20260929 16:31:32.111508 32617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:32.116621 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 32617
I20260929 16:31:32.117090 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:37259
--local_ip_for_outbound_sockets=127.23.44.3
--tserver_master_addrs=127.23.44.62:42249
--webserver_port=34489
--webserver_interface=127.23.44.3
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.44.3
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:31:32.122696 32615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:32.113382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:37413 (local address 127.23.44.62:42249)
0929 16:31:32.113517 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:32.113520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:32.114526 (+ 1006us) server_negotiation.cc:407] Connection header received
0929 16:31:32.115403 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:32.115406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:32.115467 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:32.115568 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:32.116861 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.117582 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.119023 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.119254 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.119365 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.122578 (+ 3213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:37413: BlockingRecv error: recv got EOF from 127.23.44.2:37413 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:31:32.123324 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:32.113773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:37413)
0929 16:31:32.114391 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:32.114427 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:32.115206 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:32.115765 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:32.115771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:32.116094 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:32.116671 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.116693 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.117709 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.117716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.118901 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.118910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.119398 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.119402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.119557 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.120142 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:32.120159 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:32.122464 (+ 2305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":109,"threads_started":1}
W20260929 16:31:32.123752 32746 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:42249 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.44.62:42249: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:32.129060 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:32.124256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:57883)
0929 16:31:32.124375 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:32.124390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:32.124467 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:32.124732 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:32.124735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:32.124883 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:32.124962 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.124966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.125726 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.125732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.126253 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.126260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.126691 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.126695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.126789 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.127257 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:32.127268 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:32.128936 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:31:32.129364 32615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:32.124314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:57883 (local address 127.23.44.62:42249)
0929 16:31:32.124427 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:32.124431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:32.124445 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:31:32.124581 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:32.124584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:32.124635 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:32.124719 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:32.125069 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.125627 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.126364 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.126585 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.126691 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.129044 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:57883: BlockingRecv error: recv got EOF from 127.23.44.2:57883 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:32.134919 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:32.129556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:58887)
0929 16:31:32.129692 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:32.129725 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:32.129796 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:32.130153 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:32.130155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:32.130278 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:32.130341 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.130345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.131443 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.131446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.131972 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.131980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.132566 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.132568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.132622 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.133166 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:32.133178 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:32.134841 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260929 16:31:32.135097 32615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:32.129789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:58887 (local address 127.23.44.62:42249)
0929 16:31:32.129924 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:32.129929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:32.129943 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:31:32.129979 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:32.129982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:32.130021 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:32.130108 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:32.130451 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.131315 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.132083 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.132463 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.132566 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.134986 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:58887: BlockingRecv error: recv got EOF from 127.23.44.2:58887 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 16:31:32.135210 32746 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:31:32.230518 32749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:32.230754 32749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:32.230794 32749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:32.235035 32749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:32.235092 32749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:32.235148 32749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:32.240155 32749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/krb5kdc/oryx_127.23.44.3.keytab
--principal=oryx/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:37259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=34489
--tserver_master_addrs=127.23.44.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:32.241010 32749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:32.241967 32749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:32.250789 32754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:32.250815 32755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:32.250818 32757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:32.251003 32749 server_base.cc:1034] running on GCE node
I20260929 16:31:32.251395 32749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:32.251967 32749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:32.253121 32749 hybrid_clock.cc:648] HybridClock initialized: now 1790699492253102 us; error 30 us; skew 500 ppm
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699492, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:32.255807 32749 init.cc:377] Logged in from keytab as oryx/127.23.44.3@KRBTEST.COM (short username oryx)
I20260929 16:31:32.256573 32749 webserver.cc:460] Webserver started at http://127.23.44.3:34489/ using document root <none> and password file <none>
I20260929 16:31:32.257093 32749 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:32.257193 32749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:32.260910 32749 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:32.262919 32764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:32.264206 32749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:31:32.264348 32749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "235d8f511d2e4c1d804c41e517e117b0"
format_stamp: "Formatted at 2026-09-29 16:31:31 on dist-test-slave-mlmz"
server_key: "85e40f97dc13080c97ccb37b71cf569a"
server_key_iv: "d1741b004bc61e011b447092738f9125"
server_key_version: "encryptionkey@0"
I20260929 16:31:32.264973 32749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:32.295655 32749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:32.298805 32749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:32.298956 32749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:32.299467 32749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:32.300215 32749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:32.300261 32749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:32.300300 32749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:32.300315 32749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:32.309428 32749 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:37259
I20260929 16:31:32.309453 409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:37259 every 8 connection(s)
I20260929 16:31:32.310344 32749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:32.313493 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 32749
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:31:32.321317 32615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:32.312040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:36361 (local address 127.23.44.62:42249)
0929 16:31:32.312178 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:32.312181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:32.312952 (+ 771us) server_negotiation.cc:407] Connection header received
0929 16:31:32.314611 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:32.314614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:32.314666 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:32.314758 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:32.315836 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.316343 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.317505 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.317716 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.317833 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.321143 (+ 3310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:36361: BlockingRecv error: recv got EOF from 127.23.44.3:36361 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:31:32.321902 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:32.312368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:36361)
0929 16:31:32.312811 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:32.312850 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:32.314365 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:32.314921 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:32.314927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:32.315265 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:32.315692 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.315704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.316465 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.316469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.317386 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.317395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.317866 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.317869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.318026 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.318542 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:32.318557 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:32.321016 (+ 2459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
W20260929 16:31:32.322418 410 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:42249 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.44.62:42249: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
I20260929 16:31:32.323107 414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:32.314402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47830 (local address 127.23.44.62:42249)
0929 16:31:32.314643 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:32.314646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:32.314656 (+ 10us) server_negotiation.cc:407] Connection header received
0929 16:31:32.314686 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:32.314688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:32.314726 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:32.314826 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:32.315512 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.316041 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.316888 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.317121 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.317239 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.318749 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:32.318755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:32.318771 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:32.318805 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:32.320828 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:32.321326 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:32.321332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:32.321338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:32.321494 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:32.321761 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:32.321765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:32.321766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:32.322180 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:32.322378 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:32.322717 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 16:31:32.322939 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:32.327737 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:32.322847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:55685)
0929 16:31:32.323060 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:32.323073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:32.323147 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:32.323384 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:32.323387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:32.323538 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:32.323640 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.323646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.324334 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.324337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.324995 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.325008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.325509 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.325512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.325606 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.325957 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:32.325968 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:32.327609 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153}
W20260929 16:31:32.327821 32615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:32.323055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:55685 (local address 127.23.44.62:42249)
0929 16:31:32.323173 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:32.323178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:32.323194 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:31:32.323235 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:32.323239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:32.323278 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:32.323352 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:32.323751 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.324214 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.325114 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.325401 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.325509 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.327723 (+ 2214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:55685: BlockingRecv error: recv got EOF from 127.23.44.3:55685 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:32.332530 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:32.328176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:60185)
0929 16:31:32.328284 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:32.328299 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:32.328371 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:32.328577 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:32.328580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:32.328732 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:32.328792 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.328797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.329529 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.329535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.330035 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.330043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.330473 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.330476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.330552 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.330937 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:32.330947 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:32.332411 (+ 1464us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:31:32.332607 32615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:32.328257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:60185 (local address 127.23.44.62:42249)
0929 16:31:32.328385 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:32.328391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:32.328408 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:31:32.328442 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:32.328445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:32.328476 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:32.328565 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:32.328896 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.329431 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.330147 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.330368 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.330474 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.332506 (+ 2032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:60185: BlockingRecv error: recv got EOF from 127.23.44.3:60185 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 16:31:32.332718 410 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:32 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:32.957166 415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:32.951463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:40645)
0929 16:31:32.951804 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:32.951819 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:32.951912 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:32.952177 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:32.952179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:32.952359 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:32.952457 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.952462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.953245 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.953249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.953957 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:32.953967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.954411 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.954415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:32.954485 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.955058 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:32.955069 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:32.956989 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":131,"threads_started":1}
W20260929 16:31:32.957247 416 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:32.951591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:40645 (local address 127.23.44.62:42249)
0929 16:31:32.951852 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:32.951856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:32.951868 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:31:32.952001 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:32.952003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:32.952044 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:32.952132 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:32.952560 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.953123 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.954062 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:32.954291 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:32.954370 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:32.957108 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:40645: BlockingRecv error: recv got EOF from 127.23.44.1:40645 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
Sep 29 16:31:33 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:33 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:33.141777 417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:33.135887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:34091)
0929 16:31:33.136147 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:33.136161 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:33.136254 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:33.136485 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:33.136488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:33.136697 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:33.136813 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:33.136819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:33.137666 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:33.137669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:33.138227 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:33.138236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:33.138820 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:33.138823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:33.138896 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:33.139396 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:33.139411 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:33.141612 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
W20260929 16:31:33.141923 416 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:33.136018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:34091 (local address 127.23.44.62:42249)
0929 16:31:33.136150 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:33.136154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:33.136236 (+ 82us) server_negotiation.cc:407] Connection header received
0929 16:31:33.136332 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:33.136334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:33.136378 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:33.136449 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:33.136928 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:33.137536 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:33.138490 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:33.138703 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:33.138817 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:33.141780 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:34091: BlockingRecv error: recv got EOF from 127.23.44.2:34091 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 16:31:33 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:33 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:33.339149 418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:33.333344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:37157)
0929 16:31:33.333745 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:33.333763 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:33.333887 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:33.334177 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:33.334182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:33.334444 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:33.334612 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:33.334622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:33.335514 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:33.335516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:33.336100 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:33.336108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:33.336526 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:33.336528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:33.336595 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:33.337175 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:33.337195 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:33.339033 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":86,"threads_started":1}
W20260929 16:31:33.339272 416 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:33.333500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:37157 (local address 127.23.44.62:42249)
0929 16:31:33.333660 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:33.333665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:33.333866 (+ 201us) server_negotiation.cc:407] Connection header received
0929 16:31:33.333976 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:33.333979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:33.334038 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:33.334108 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:33.334743 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:33.335397 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:33.336223 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:33.336432 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:33.336509 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:33.339153 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:37157: BlockingRecv error: recv got EOF from 127.23.44.3:37157 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 16:31:33 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:33 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:33.963402 419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:33.958046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:45201)
0929 16:31:33.958324 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:33.958338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:33.958426 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:33.958649 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:33.958651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:33.958855 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:33.958953 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:33.958958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:33.959719 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:33.959722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:33.960288 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:33.960297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:33.960730 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:33.960732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:33.960799 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:33.961436 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:33.961449 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:33.963268 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
W20260929 16:31:33.963558 420 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:33.958115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:45201 (local address 127.23.44.62:42249)
0929 16:31:33.958369 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:33.958372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:33.958385 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:31:33.958506 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:33.958508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:33.958548 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:33.958620 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:33.959059 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:33.959592 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:33.960408 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:33.960622 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:33.960703 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:33.963414 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:45201: BlockingRecv error: recv got EOF from 127.23.44.1:45201 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":60,"threads_started":1}
Sep 29 16:31:34 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:34 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:34.147522 421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:34.142466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:37079)
0929 16:31:34.142768 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:34.142781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:34.142874 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:34.143098 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:34.143100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:34.143269 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:34.143372 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:34.143378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:34.144095 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:34.144098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:34.144682 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:34.144691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:34.145082 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:34.145085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:34.145143 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:34.145662 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:34.145673 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:34.147401 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":84,"threads_started":1}
W20260929 16:31:34.147689 420 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:34.142576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:37079 (local address 127.23.44.62:42249)
0929 16:31:34.142708 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:34.142719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:34.142842 (+ 123us) server_negotiation.cc:407] Connection header received
0929 16:31:34.142946 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:34.142948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:34.142988 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:34.143054 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:34.143470 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:34.143991 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:34.144789 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:34.144990 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:34.145060 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:34.147554 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:37079: BlockingRecv error: recv got EOF from 127.23.44.2:37079 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 16:31:34 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:34 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:34.345541 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:34.339942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:37979)
0929 16:31:34.340265 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:34.340281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:34.340425 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:34.340705 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:34.340708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:34.340916 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:34.341001 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:34.341007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:34.341852 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:34.341858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:34.342434 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:34.342442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:34.342903 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:34.342907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:34.342984 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:34.343488 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:34.343499 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:34.345350 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":99,"threads_started":1}
W20260929 16:31:34.345616 420 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:34.340097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:37979 (local address 127.23.44.62:42249)
0929 16:31:34.340260 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:34.340265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:34.340371 (+ 106us) server_negotiation.cc:407] Connection header received
0929 16:31:34.340533 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:34.340538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:34.340592 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:34.340687 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:34.341119 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:34.341739 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:34.342558 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:34.342789 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:34.342903 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:34.345488 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:37979: BlockingRecv error: recv got EOF from 127.23.44.3:37979 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 16:31:34 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:34 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:34.974586 423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:34.964226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:57171)
0929 16:31:34.964534 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:34.964548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:34.964634 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:34.964932 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:34.964934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:34.965134 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:34.965236 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:34.965242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:34.966147 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:34.966150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:34.966729 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:34.966738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:34.967207 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:34.967211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:34.967305 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:34.972391 (+ 5086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:34.972407 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:34.974451 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":117,"threads_started":1}
W20260929 16:31:34.974812 424 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:34.964283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:57171 (local address 127.23.44.62:42249)
0929 16:31:34.964586 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:34.964589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:34.964613 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:31:34.964720 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:34.964722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:34.964765 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:34.964846 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:34.965472 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:34.966016 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:34.966854 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:34.967086 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:34.967191 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:34.974610 (+ 7419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:57171: BlockingRecv error: recv got EOF from 127.23.44.1:57171 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":99,"threads_started":1}
Sep 29 16:31:35 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:35 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:35.153793 425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:35.148367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:33653)
0929 16:31:35.148643 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:35.148656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:35.148752 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:35.148984 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:35.148987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:35.149163 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:35.149292 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:35.149297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:35.149986 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:35.149988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:35.150546 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:35.150553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:35.150926 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:35.150929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:35.150989 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:35.151442 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:35.151456 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:35.153677 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":104,"threads_started":1}
W20260929 16:31:35.153944 424 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:35.148411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:33653 (local address 127.23.44.62:42249)
0929 16:31:35.148552 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:35.148556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:35.148725 (+ 169us) server_negotiation.cc:407] Connection header received
0929 16:31:35.148831 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:35.148833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:35.148878 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:35.148949 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:35.149384 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:35.149898 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:35.150636 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:35.150837 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:35.150908 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:35.153827 (+ 2919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:33653: BlockingRecv error: recv got EOF from 127.23.44.2:33653 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 16:31:35 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:35 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:35.351838 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:35.346286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:36005)
0929 16:31:35.346542 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:35.346555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:35.346650 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:35.346889 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:35.346891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:35.347053 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:35.347148 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:35.347153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:35.347891 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:35.347894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:35.348757 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:35.348769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:35.349332 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:35.349335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:35.349410 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:35.349905 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:35.349916 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:35.351729 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
W20260929 16:31:35.351908 424 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:35.346381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:36005 (local address 127.23.44.62:42249)
0929 16:31:35.346513 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:35.346517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:35.346622 (+ 105us) server_negotiation.cc:407] Connection header received
0929 16:31:35.346738 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:35.346740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:35.346783 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:35.346851 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:35.347254 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:35.347779 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:35.348892 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:35.349228 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:35.349326 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:35.351820 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:36005: BlockingRecv error: recv got EOF from 127.23.44.3:36005 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 16:31:35 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:35 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:35.981624 427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:35.975430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:56107)
0929 16:31:35.975710 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:35.975735 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:35.975848 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:35.976173 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:35.976176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:35.976384 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:35.976520 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:35.976528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:35.977721 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:35.977724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:35.978288 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:35.978296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:35.978818 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:35.978821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:35.978899 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:35.979429 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:35.979440 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:35.981477 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":127,"threads_started":1}
W20260929 16:31:35.981889 428 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:35.975528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:56107 (local address 127.23.44.62:42249)
0929 16:31:35.975806 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:35.975813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:35.975833 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:31:35.975983 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:35.975986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:35.976031 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:35.976139 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:35.976677 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:35.977593 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:35.978426 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:35.978668 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:35.978783 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:35.981745 (+ 2962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:56107: BlockingRecv error: recv got EOF from 127.23.44.1:56107 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
Sep 29 16:31:36 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:36 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:36.160279 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:36.154733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:57059)
0929 16:31:36.155005 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:36.155017 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:36.155153 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:36.155422 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:36.155425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:36.155629 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:36.155720 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:36.155726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:36.156505 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:36.156509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:36.157068 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:36.157076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:36.157587 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:36.157590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:36.157679 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:36.158241 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:36.158253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:36.160113 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
W20260929 16:31:36.160357 428 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:36.154779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:57059 (local address 127.23.44.62:42249)
0929 16:31:36.154951 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:36.154956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:36.155094 (+ 138us) server_negotiation.cc:407] Connection header received
0929 16:31:36.155262 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:36.155265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:36.155315 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:36.155401 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:36.155862 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:36.156384 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:36.157188 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:36.157471 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:36.157584 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:36.160245 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:57059: BlockingRecv error: recv got EOF from 127.23.44.2:57059 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 16:31:36 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:36 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:36.359472 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:36.352626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:55583)
0929 16:31:36.352943 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:36.352958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:36.353063 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:36.353463 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:36.353465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:36.353684 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:36.353783 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:36.353789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:36.354706 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:36.354720 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:36.355384 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:36.355400 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:36.356006 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:36.356009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:36.356134 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:36.356823 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:36.356836 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:36.359338 (+ 2502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":89,"threads_started":1}
W20260929 16:31:36.359947 428 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:36.352703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:55583 (local address 127.23.44.62:42249)
0929 16:31:36.352852 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:36.352859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:36.353029 (+ 170us) server_negotiation.cc:407] Connection header received
0929 16:31:36.353146 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:36.353148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:36.353193 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:36.353283 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:36.353897 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:36.354425 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:36.355583 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:36.355850 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:36.355944 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:36.359726 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:55583: BlockingRecv error: recv got EOF from 127.23.44.3:55583 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 16:31:36 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:36 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:36.988688 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:36.982453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:58563)
0929 16:31:36.982796 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:36.982810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:36.982935 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:36.983320 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:36.983323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:36.983624 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:36.983760 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:36.983776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:36.984682 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:36.984688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:36.985421 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:36.985431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:36.985932 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:36.985936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:36.986023 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:36.986702 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:36.986719 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:36.988519 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":77,"threads_started":1}
W20260929 16:31:36.988772 432 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:36.982554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:58563 (local address 127.23.44.62:42249)
0929 16:31:36.982876 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:36.982883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:36.982902 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:31:36.983100 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:36.983103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:36.983155 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:36.983294 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:36.983903 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:36.984563 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:36.985557 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:36.985806 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:36.985929 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:36.988642 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:58563: BlockingRecv error: recv got EOF from 127.23.44.1:58563 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":74,"threads_started":1}
Sep 29 16:31:37 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:37 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:37.167728 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:37.161197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:33413)
0929 16:31:37.161539 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:37.161555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:37.161650 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:37.161905 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:37.161908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:37.162127 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:37.162227 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:37.162233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:37.163022 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:37.163025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:37.163761 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:37.163771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:37.164631 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:37.164634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:37.164707 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:37.165290 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:37.165304 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:37.167532 (+ 2228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":156,"threads_started":1}
W20260929 16:31:37.167826 432 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:37.161412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:33413 (local address 127.23.44.62:42249)
0929 16:31:37.161564 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:37.161568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:37.161608 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:31:37.161739 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:37.161741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:37.161789 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:37.161869 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:37.162369 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:37.162890 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:37.163978 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:37.164453 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:37.164612 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:37.167662 (+ 3050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:33413: BlockingRecv error: recv got EOF from 127.23.44.2:33413 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 16:31:37 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:37 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:37.366534 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:37.360401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:37555)
0929 16:31:37.360734 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:37.360747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:37.360837 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:37.361118 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:37.361121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:37.361390 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:37.361504 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:37.361509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:37.362273 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:37.362276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:37.362869 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:37.362877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:37.363324 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:37.363326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:37.363385 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:37.363951 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:37.363970 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:37.366397 (+ 2427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
W20260929 16:31:37.366689 432 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:37.360424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:37555 (local address 127.23.44.62:42249)
0929 16:31:37.360588 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:37.360593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:37.360803 (+ 210us) server_negotiation.cc:407] Connection header received
0929 16:31:37.360928 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:37.360931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:37.360996 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:37.361064 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:37.361611 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:37.362134 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:37.363011 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:37.363222 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:37.363311 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:37.366553 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:37555: BlockingRecv error: recv got EOF from 127.23.44.3:37555 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 16:31:37 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:37 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:37.995337 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:37.989639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:54573)
0929 16:31:37.989943 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:37.989958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:37.990055 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:37.990330 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:37.990332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:37.990519 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:37.990616 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:37.990621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:37.991482 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:37.991484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:37.992076 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:37.992084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:37.992577 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:37.992580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:37.992684 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:37.993321 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:37.993335 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:37.995225 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":70,"threads_started":1}
W20260929 16:31:37.995579 436 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:37.989688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:54573 (local address 127.23.44.62:42249)
0929 16:31:37.989989 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:37.989992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:37.990005 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:31:37.990152 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:37.990154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:37.990196 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:37.990285 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:37.990715 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:37.991392 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:37.992174 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:37.992476 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:37.992556 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:37.995436 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:54573: BlockingRecv error: recv got EOF from 127.23.44.1:54573 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":131,"threads_started":1}
Sep 29 16:31:38 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:38 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:38.174014 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:38.168585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:42895)
0929 16:31:38.168848 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:38.168865 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:38.168965 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:38.169273 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:38.169276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:38.169484 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:38.169575 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:38.169580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:38.170309 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:38.170312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:38.170908 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:38.170924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:38.171390 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:38.171393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:38.171459 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:38.172053 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:38.172072 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:38.173896 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":64,"threads_started":1}
W20260929 16:31:38.174188 436 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:38.168727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:42895 (local address 127.23.44.62:42249)
0929 16:31:38.168884 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:38.168887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:38.168923 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:31:38.169068 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:38.169071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:38.169131 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:38.169207 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:38.169676 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:38.170195 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:38.171055 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:38.171277 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:38.171360 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:38.174030 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:42895: BlockingRecv error: recv got EOF from 127.23.44.2:42895 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 16:31:38 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:38 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:38.373116 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:38.367375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:56965)
0929 16:31:38.367644 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:38.367658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:38.367816 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:38.368037 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:38.368039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:38.368219 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:38.368307 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:38.368312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:38.369156 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:38.369158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:38.369784 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:38.369793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:38.370199 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:38.370202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:38.370265 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:38.370843 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:38.370882 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:38.373001 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
W20260929 16:31:38.373250 436 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:38.367468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:56965 (local address 127.23.44.62:42249)
0929 16:31:38.367669 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:38.367680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:38.367778 (+ 98us) server_negotiation.cc:407] Connection header received
0929 16:31:38.367891 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:38.367894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:38.367938 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:38.368006 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:38.368460 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:38.369055 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:38.369895 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:38.370103 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:38.370191 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:38.373127 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:56965: BlockingRecv error: recv got EOF from 127.23.44.3:56965 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 16:31:38 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:38 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:39.002034 439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:38.996092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:45035)
0929 16:31:38.996419 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:38.996433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:38.996521 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:38.997182 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:38.997185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:38.997383 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:38.997495 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:38.997500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:38.998343 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:38.998345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:38.998917 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:38.998924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:38.999358 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:38.999361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:38.999424 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:38.999932 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:38.999943 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:39.001917 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
W20260929 16:31:39.002202 440 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:38.996503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:45035 (local address 127.23.44.62:42249)
0929 16:31:38.996848 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:38.996852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:38.996870 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:31:38.996932 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:38.996935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:38.996992 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:38.997119 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:38.997657 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:38.998233 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:38.999029 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:38.999252 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:38.999341 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:39.002064 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:45035: BlockingRecv error: recv got EOF from 127.23.44.1:45035 (error 108)
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":133,"threads_started":1}
Sep 29 16:31:39 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:39 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:39.180414 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:39.174799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:47479)
0929 16:31:39.175167 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:39.175180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:39.175284 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:39.175553 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:39.175555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:39.175769 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:39.175860 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:39.175865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:39.176640 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:39.176643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:39.177231 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:39.177239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:39.177766 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:39.177769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:39.177830 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:39.178334 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:39.178346 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:39.180247 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":98,"threads_started":1}
W20260929 16:31:39.180728 440 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:39.174944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:47479 (local address 127.23.44.62:42249)
0929 16:31:39.175117 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:39.175120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:39.175268 (+ 148us) server_negotiation.cc:407] Connection header received
0929 16:31:39.175371 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:39.175373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:39.175421 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:39.175488 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:39.175990 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:39.176535 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:39.177416 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:39.177664 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:39.177748 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:39.180384 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:47479: BlockingRecv error: recv got EOF from 127.23.44.2:47479 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 16:31:39 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:39 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:39.382445 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:39.374302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:53223)
0929 16:31:39.374752 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:39.374775 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:39.374946 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:39.375312 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:39.375314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:39.375589 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:39.375714 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:39.375720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:39.376738 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:39.376742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:39.377818 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:39.377832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:39.378677 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:39.378681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:39.378776 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:39.379733 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:39.379751 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:39.382266 (+ 2515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":151,"threads_started":1}
W20260929 16:31:39.382532 440 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:39.374601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:53223 (local address 127.23.44.62:42249)
0929 16:31:39.374883 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:39.374888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:39.374909 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:31:39.375093 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:39.375096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:39.375158 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:39.375261 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:39.375935 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:39.376577 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:39.378026 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:39.378488 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:39.378603 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:39.382412 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:53223: BlockingRecv error: recv got EOF from 127.23.44.3:53223 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Sep 29 16:31:40 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:40 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:40.013438 443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:40.002910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:46523)
0929 16:31:40.003192 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:40.003207 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:40.003305 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:40.003869 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:40.003872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:40.004101 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:40.004198 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:40.004204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:40.004986 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:40.004991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:40.005590 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:40.005599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:40.006062 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:40.006066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:40.006163 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:40.010558 (+ 4395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:40.010571 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:40.013231 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
W20260929 16:31:40.013617 444 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:40.003294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:46523 (local address 127.23.44.62:42249)
0929 16:31:40.003633 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:40.003637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:40.003650 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:31:40.003686 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:40.003688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:40.003730 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:40.003841 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:40.004329 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:40.004872 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:40.005716 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:40.005943 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:40.006056 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:40.013408 (+ 7352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:46523: BlockingRecv error: recv got EOF from 127.23.44.1:46523 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
Sep 29 16:31:40 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:40 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:40.187289 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:40.181521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:55427)
0929 16:31:40.181863 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:40.181875 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:40.182017 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:40.182365 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:40.182368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:40.182624 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:40.182727 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:40.182733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:40.183702 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:40.183705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:40.184261 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:40.184269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:40.184753 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:40.184755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:40.184815 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:40.185362 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:40.185373 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:40.187167 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":136,"threads_started":1}
W20260929 16:31:40.187515 444 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:40.181669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:55427 (local address 127.23.44.62:42249)
0929 16:31:40.181827 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:40.181832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:40.182043 (+ 211us) server_negotiation.cc:407] Connection header received
0929 16:31:40.182133 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:40.182135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:40.182184 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:40.182323 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:40.182848 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:40.183566 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:40.184387 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:40.184650 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:40.184741 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:40.187377 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:55427: BlockingRecv error: recv got EOF from 127.23.44.2:55427 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 16:31:40 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:40 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:40.389045 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:40.383362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:37177)
0929 16:31:40.383690 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:40.383702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:40.383832 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:40.384126 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:40.384129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:40.384339 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:40.384427 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:40.384433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:40.385181 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:40.385187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:40.385785 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:40.385793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:40.386236 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:40.386239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:40.386322 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:40.386872 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:40.386883 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:40.388917 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":86,"threads_started":1}
W20260929 16:31:40.389442 444 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:40.383409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:37177 (local address 127.23.44.62:42249)
0929 16:31:40.383572 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:40.383577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:40.383780 (+ 203us) server_negotiation.cc:407] Connection header received
0929 16:31:40.383938 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:40.383942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:40.384013 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:40.384103 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:40.384558 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:40.385074 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:40.385903 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:40.386127 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:40.386235 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:40.389329 (+ 3094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:37177: BlockingRecv error: recv got EOF from 127.23.44.3:37177 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 16:31:41 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:41 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:41.020004 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:41.014321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:41755)
0929 16:31:41.014694 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:41.014708 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:41.014802 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:41.015042 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:41.015044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:41.015221 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:41.015315 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:41.015320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:41.016137 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:41.016140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:41.016707 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:41.016715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:41.017163 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:41.017166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:41.017231 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:41.017819 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:41.017831 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:41.019892 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
W20260929 16:31:41.020226 448 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:41.014485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:41755 (local address 127.23.44.62:42249)
0929 16:31:41.014739 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:41.014743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:41.014758 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:31:41.014872 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:41.014875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:41.014918 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:41.015000 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:41.015422 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:41.016025 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:41.016812 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:41.017040 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:41.017122 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:41.020036 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:41755: BlockingRecv error: recv got EOF from 127.23.44.1:41755 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":65,"threads_started":1}
Sep 29 16:31:41 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:41 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:41.195199 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:41.188075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:54473)
0929 16:31:41.188333 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:41.188355 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:41.188441 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:41.188814 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:41.188816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:41.188988 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:41.189087 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:41.189093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:41.190016 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:41.190019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:41.190670 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:41.190680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:41.191359 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:41.191364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:41.191454 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:41.192362 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:41.192375 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:41.195075 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
W20260929 16:31:41.195444 448 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:41.188371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:54473 (local address 127.23.44.62:42249)
0929 16:31:41.188552 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:41.188556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:41.188573 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:31:41.188624 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:41.188627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:41.188678 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:41.188782 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:41.189239 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:41.189869 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:41.190844 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:41.191112 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:41.191383 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:41.195298 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:54473: BlockingRecv error: recv got EOF from 127.23.44.2:54473 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 16:31:41 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:41 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:41.395352 450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:41.389839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:34779)
0929 16:31:41.390117 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:41.390130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:41.390226 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:41.390445 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:41.390447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:41.390633 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:41.390726 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:41.390731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:41.391576 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:41.391579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:41.392243 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:41.392273 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:41.392696 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:41.392698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:41.392759 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:41.393309 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:41.393325 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:41.395240 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
W20260929 16:31:41.395566 448 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:41.389892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:34779 (local address 127.23.44.62:42249)
0929 16:31:41.390032 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:41.390037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:41.390198 (+ 161us) server_negotiation.cc:407] Connection header received
0929 16:31:41.390301 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:41.390303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:41.390350 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:41.390415 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:41.390849 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:41.391464 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:41.392373 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:41.392584 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:41.392673 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:41.395436 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:34779: BlockingRecv error: recv got EOF from 127.23.44.3:34779 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 16:31:42 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:42 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:42.026239 451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:42.020843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:33539)
0929 16:31:42.021170 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:42.021184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:42.021300 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:42.021520 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:42.021522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:42.021707 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:42.021798 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:42.021803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:42.022515 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:42.022517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:42.023073 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:42.023080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:42.023504 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:42.023507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:42.023571 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:42.024128 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:42.024141 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:42.026121 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
W20260929 16:31:42.026569 452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:42.020907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:33539 (local address 127.23.44.62:42249)
0929 16:31:42.021186 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:42.021189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:42.021235 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:31:42.021377 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:42.021379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:42.021416 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:42.021484 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:42.021895 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:42.022420 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:42.023179 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:42.023401 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:42.023478 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:42.026396 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:33539: BlockingRecv error: recv got EOF from 127.23.44.1:33539 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":108,"threads_started":1}
Sep 29 16:31:42 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:42 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:42.201613 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:42.196001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:41787)
0929 16:31:42.196249 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:42.196262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:42.196350 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:42.196556 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:42.196558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:42.196778 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:42.196922 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:42.196930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:42.198083 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:42.198086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:42.198640 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:42.198648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:42.199105 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:42.199108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:42.199173 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:42.199647 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:42.199658 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:42.201469 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
W20260929 16:31:42.201819 452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:42.196084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:41787 (local address 127.23.44.62:42249)
0929 16:31:42.196242 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:42.196246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:42.196313 (+ 67us) server_negotiation.cc:407] Connection header received
0929 16:31:42.196414 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:42.196416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:42.196456 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:42.196523 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:42.197047 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:42.197987 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:42.198750 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:42.198975 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:42.199056 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:42.201638 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:41787: BlockingRecv error: recv got EOF from 127.23.44.2:41787 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 16:31:42 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:42 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:42.402415 454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:42.396132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:46925)
0929 16:31:42.396442 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:42.396460 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:42.396580 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:42.396861 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:42.396864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:42.397077 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:42.397168 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:42.397174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:42.397967 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:42.397970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:42.398524 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:42.398532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:42.398975 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:42.398978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:42.399069 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:42.399651 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:42.399663 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:42.402299 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
W20260929 16:31:42.402659 452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:42.396304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:46925 (local address 127.23.44.62:42249)
0929 16:31:42.396450 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:42.396455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:42.396557 (+ 102us) server_negotiation.cc:407] Connection header received
0929 16:31:42.396693 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:42.396697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:42.396750 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:42.396836 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:42.397323 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:42.397845 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:42.398643 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:42.398861 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:42.398971 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:42.402550 (+ 3579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:46925: BlockingRecv error: recv got EOF from 127.23.44.3:46925 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 16:31:43 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:43 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:43.032335 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:43.027010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:54229)
0929 16:31:43.027305 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:43.027318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:43.027418 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:43.027632 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:43.027634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:43.027807 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:43.027899 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:43.027904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:43.028597 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:43.028599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:43.029131 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:43.029137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:43.029622 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:43.029625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:43.029684 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:43.030199 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:43.030211 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:43.032198 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
W20260929 16:31:43.032523 456 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:43.027080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:54229 (local address 127.23.44.62:42249)
0929 16:31:43.027322 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:43.027326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:43.027370 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:31:43.027483 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:43.027485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:43.027524 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:43.027596 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:43.027994 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:43.028511 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:43.029272 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:43.029495 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:43.029612 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:43.032349 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:54229: BlockingRecv error: recv got EOF from 127.23.44.1:54229 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":61,"threads_started":1}
Sep 29 16:31:43 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:43 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:43.208024 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:43.202530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:54951)
0929 16:31:43.202816 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:43.202831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:43.202931 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:43.203167 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:43.203170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:43.203358 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:43.203454 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:43.203459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:43.204207 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:43.204210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:43.204815 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:43.204823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:43.205242 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:43.205245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:43.205382 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:43.205866 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:43.205878 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:43.207877 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":100,"threads_started":1}
W20260929 16:31:43.208216 456 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:43.202601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:54951 (local address 127.23.44.62:42249)
0929 16:31:43.202763 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:43.202766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:43.202939 (+ 173us) server_negotiation.cc:407] Connection header received
0929 16:31:43.203005 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:43.203007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:43.203053 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:43.203135 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:43.203561 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:43.204088 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:43.204932 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:43.205145 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:43.205221 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:43.208065 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:54951: BlockingRecv error: recv got EOF from 127.23.44.2:54951 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 16:31:43 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:43 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:43.408769 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:43.403161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:34099)
0929 16:31:43.403419 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:43.403435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:43.403542 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:43.403806 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:43.403808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:43.403981 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:43.404075 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:43.404080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:43.404801 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:43.404804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:43.405648 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:43.405658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:43.406245 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:43.406248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:43.406308 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:43.406824 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:43.406835 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:43.408658 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
W20260929 16:31:43.408905 456 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:43.403295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:34099 (local address 127.23.44.62:42249)
0929 16:31:43.403432 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:43.403437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:43.403520 (+ 83us) server_negotiation.cc:407] Connection header received
0929 16:31:43.403653 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:43.403655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:43.403701 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:43.403774 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:43.404174 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:43.404688 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:43.405789 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:43.406135 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:43.406217 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:43.408789 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:34099: BlockingRecv error: recv got EOF from 127.23.44.3:34099 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 16:31:44 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:44 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:44.039077 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:44.033094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:37699)
0929 16:31:44.033371 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:44.033385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:44.033471 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:44.033912 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:44.033915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:44.034108 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:44.034219 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:44.034225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:44.035138 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:44.035143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:44.035985 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:44.035994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:44.036477 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:44.036481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:44.036544 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:44.037074 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:44.037092 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:44.038957 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":93,"threads_started":1}
W20260929 16:31:44.039202 460 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:44.033367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:37699 (local address 127.23.44.62:42249)
0929 16:31:44.033639 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:44.033643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:44.033656 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:31:44.033693 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:44.033696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:44.033747 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:44.033833 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:44.034337 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:44.034978 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:44.036115 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:44.036375 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:44.036462 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:44.039073 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:37699: BlockingRecv error: recv got EOF from 127.23.44.1:37699 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
Sep 29 16:31:44 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:44 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:44.214195 461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:44.209074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:50781)
0929 16:31:44.209390 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:44.209403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:44.209497 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:44.209726 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:44.209729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:44.209903 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:44.209995 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:44.210000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:44.210706 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:44.210709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:44.211273 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:44.211280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:44.211698 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:44.211700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:44.211758 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:44.212221 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:44.212232 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:44.214084 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
W20260929 16:31:44.214391 460 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:44.209121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:50781 (local address 127.23.44.62:42249)
0929 16:31:44.209298 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:44.209302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:44.209477 (+ 175us) server_negotiation.cc:407] Connection header received
0929 16:31:44.209563 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:44.209565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:44.209623 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:44.209690 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:44.210089 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:44.210594 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:44.211383 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:44.211589 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:44.211684 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:44.214246 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:50781: BlockingRecv error: recv got EOF from 127.23.44.2:50781 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 16:31:44 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:44 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:44.415225 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:44.409651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:33135)
0929 16:31:44.410016 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:44.410042 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:44.410139 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:44.410362 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:44.410364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:44.410575 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:44.410673 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:44.410679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:44.411444 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:44.411446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:44.412059 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:44.412067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:44.412478 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:44.412480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:44.412546 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:44.413134 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:44.413146 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:44.415058 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":197,"threads_started":1}
W20260929 16:31:44.415261 460 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:44.409758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:33135 (local address 127.23.44.62:42249)
0929 16:31:44.409913 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:44.409916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:44.410097 (+ 181us) server_negotiation.cc:407] Connection header received
0929 16:31:44.410207 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:44.410209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:44.410254 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:44.410328 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:44.410807 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:44.411333 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:44.412173 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:44.412379 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:44.412459 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:44.415163 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:33135: BlockingRecv error: recv got EOF from 127.23.44.3:33135 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 16:31:45 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:45 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:45.049650 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:45.039893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:49773)
0929 16:31:45.040165 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:45.040177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:45.040263 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:45.040519 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:45.040522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:45.040723 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:45.040819 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:45.040824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:45.041652 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:45.041659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:45.042212 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:45.042220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:45.042660 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:45.042663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:45.042729 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:45.047593 (+ 4864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:45.047608 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:45.049489 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":109,"threads_started":1}
W20260929 16:31:45.049852 464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:45.039973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:49773 (local address 127.23.44.62:42249)
0929 16:31:45.040204 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:45.040207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:45.040219 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:31:45.040343 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:45.040345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:45.040385 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:45.040460 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:45.040920 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:45.041536 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:45.042313 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:45.042549 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:45.042625 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:45.049686 (+ 7061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:49773: BlockingRecv error: recv got EOF from 127.23.44.1:49773 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
Sep 29 16:31:45 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:45 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:45.220468 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:45.214958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:55089)
0929 16:31:45.215260 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:45.215277 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:45.215391 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:45.215678 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:45.215681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:45.215887 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:45.215972 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:45.215978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:45.216727 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:45.216733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:45.217320 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:45.217329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:45.217786 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:45.217789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:45.217872 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:45.218435 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:45.218456 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:45.220291 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":109,"threads_started":1}
W20260929 16:31:45.220513 464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:45.215111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:55089 (local address 127.23.44.62:42249)
0929 16:31:45.215262 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:45.215268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:45.215354 (+ 86us) server_negotiation.cc:407] Connection header received
0929 16:31:45.215500 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:45.215504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:45.215551 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:45.215645 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:45.216090 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:45.216619 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:45.217440 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:45.217676 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:45.217784 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:45.220408 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:55089: BlockingRecv error: recv got EOF from 127.23.44.2:55089 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 16:31:45 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:45 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:45.421770 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:45.416480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:50369)
0929 16:31:45.416808 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:45.416824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:45.416935 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:45.417229 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:45.417232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:45.417441 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:45.417532 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:45.417537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:45.418304 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:45.418307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:45.418878 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:45.418886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:45.419293 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:45.419296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:45.419358 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:45.419906 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:45.419920 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:45.421663 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":143,"threads_started":1}
W20260929 16:31:45.421904 464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:45.416459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:50369 (local address 127.23.44.62:42249)
0929 16:31:45.416625 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:45.416629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:45.416938 (+ 309us) server_negotiation.cc:407] Connection header received
0929 16:31:45.417005 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:45.417008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:45.417071 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:45.417198 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:45.417663 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:45.418175 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:45.418988 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:45.419190 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:45.419264 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:45.421788 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:50369: BlockingRecv error: recv got EOF from 127.23.44.3:50369 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 16:31:46 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:46 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:46.056043 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:46.050520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:50831)
0929 16:31:46.050778 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:46.050791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:46.050876 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:46.051124 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:46.051126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:46.051309 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:46.051414 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:46.051419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:46.052178 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:46.052181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:46.052837 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:46.052850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:46.053476 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:46.053481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:46.053587 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:46.054189 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:46.054201 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:46.055913 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
W20260929 16:31:46.056182 468 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:46.050633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:50831 (local address 127.23.44.62:42249)
0929 16:31:46.050912 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:46.050915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:46.050927 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:31:46.050960 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:46.050962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:46.051001 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:46.051087 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:46.051540 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:46.052078 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:46.052959 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:46.053353 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:46.053456 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:46.056050 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:50831: BlockingRecv error: recv got EOF from 127.23.44.1:50831 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
Sep 29 16:31:46 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:46 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:46.227016 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:46.221366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:38519)
0929 16:31:46.221656 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:46.221669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:46.221762 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:46.221994 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:46.221996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:46.222168 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:46.222259 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:46.222264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:46.223034 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:46.223037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:46.223576 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:46.223583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:46.224049 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:46.224051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:46.224145 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:46.224763 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:46.224775 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:46.226896 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":127,"threads_started":1}
W20260929 16:31:46.227211 468 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:46.221447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:38519 (local address 127.23.44.62:42249)
0929 16:31:46.221589 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:46.221593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:46.221732 (+ 139us) server_negotiation.cc:407] Connection header received
0929 16:31:46.221838 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:46.221840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:46.221884 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:46.221960 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:46.222374 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:46.222935 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:46.223693 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:46.223907 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:46.224028 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:46.227064 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:38519: BlockingRecv error: recv got EOF from 127.23.44.2:38519 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 16:31:46 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:46 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.3@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:46.427951 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:46.422575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.3:51495)
0929 16:31:46.422935 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:46.422956 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:46.423108 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:46.423383 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:46.423386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:46.423602 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:46.423690 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:46.423696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:46.424430 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:46.424435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:46.424996 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:46.425003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:46.425551 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:46.425556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:46.425671 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:46.426136 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:46.426147 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:46.427831 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":106,"threads_started":1}
W20260929 16:31:46.428128 468 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:46.422717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:51495 (local address 127.23.44.62:42249)
0929 16:31:46.422889 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:46.422894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:46.423036 (+ 142us) server_negotiation.cc:407] Connection header received
0929 16:31:46.423218 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:46.423222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:46.423275 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:46.423362 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:46.423809 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:46.424323 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:46.425118 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:46.425395 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:46.425504 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:46.427982 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.3:51495: BlockingRecv error: recv got EOF from 127.23.44.3:51495 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 16:31:47 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:47 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.1@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:47.062235 471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:47.056989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.1:53573)
0929 16:31:47.057284 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:47.057301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:47.057398 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:47.057750 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:47.057753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:47.058008 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:47.058106 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:47.058112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.058836 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.058838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:47.059382 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:47.059389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.059822 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.059825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:47.059882 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:47.060331 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:47.060342 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:47.062117 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":110,"threads_started":1}
W20260929 16:31:47.062378 472 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:47.057049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:53573 (local address 127.23.44.62:42249)
0929 16:31:47.057342 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:47.057346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:47.057358 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:31:47.057553 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:47.057559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:47.057614 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:47.057729 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:47.058211 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.058746 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.059513 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.059726 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.059807 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:47.062249 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.1:53573: BlockingRecv error: recv got EOF from 127.23.44.1:53573 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
Sep 29 16:31:47 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:31:47 dist-test-slave-mlmz krb5kdc[31882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.44.2@KRBTEST.COM for oryx/127.23.44.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:31:47.233243 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:47.228032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:42249 (local address 127.23.44.2:45089)
0929 16:31:47.228359 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:47.228373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:47.228499 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:47.228745 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:47.228747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:47.228918 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:47.229008 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:47.229012 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.229806 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.229808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:47.230361 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:47.230369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.230794 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.230797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:47.230860 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:47.231416 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:47.231428 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:47.233135 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.44.62:42249: Server oryx/127.23.44.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
W20260929 16:31:47.233388 472 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:47.228167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:45089 (local address 127.23.44.62:42249)
0929 16:31:47.228306 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:47.228311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:47.228444 (+ 133us) server_negotiation.cc:407] Connection header received
0929 16:31:47.228600 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:47.228602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:47.228645 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:47.228708 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:47.229110 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.229706 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.230479 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.230687 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.230761 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:47.233271 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.44.2:45089: BlockingRecv error: recv got EOF from 127.23.44.2:45089 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:31:47.317410 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 32465
I20260929 16:31:47.322876 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 32617
I20260929 16:31:47.327790 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 32749
I20260929 16:31:47.332515 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 32395
2026-09-29T16:31:47Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17920 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T16:31:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:47Z Disabled control of system clock
I20260929 16:31:47.361523 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:45821
--webserver_interface=127.23.44.62
--webserver_port=0
--builtin_ntp_servers=127.23.44.20:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:45821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 16:31:47.465040 477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:47.465330 477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:47.465402 477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:47.469290 477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:47.469333 477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:47.469349 477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:47.469365 477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:47.469380 477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:47.473934 477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:45821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:45821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:47.474781 477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:47.476034 477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:47.484331 485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:47.484369 483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:47.484382 482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:47.484586 477 server_base.cc:1034] running on GCE node
I20260929 16:31:47.485052 477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:47.486140 477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:47.487303 477 hybrid_clock.cc:648] HybridClock initialized: now 1790699507487270 us; error 44 us; skew 500 ppm
I20260929 16:31:47.489130 477 webserver.cc:460] Webserver started at http://127.23.44.62:37191/ using document root <none> and password file <none>
I20260929 16:31:47.489701 477 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:47.489760 477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:47.489952 477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:47.491842 477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "254c7fd5f3a24340948db9c5fc410003"
format_stamp: "Formatted at 2026-09-29 16:31:47 on dist-test-slave-mlmz"
server_key: "88d1ea7626799d9afaaecb26b81649cb"
server_key_iv: "446d1f8e58bdbee458efbebf281a1397"
server_key_version: "encryptionkey@0"
I20260929 16:31:47.492373 477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "254c7fd5f3a24340948db9c5fc410003"
format_stamp: "Formatted at 2026-09-29 16:31:47 on dist-test-slave-mlmz"
server_key: "88d1ea7626799d9afaaecb26b81649cb"
server_key_iv: "446d1f8e58bdbee458efbebf281a1397"
server_key_version: "encryptionkey@0"
I20260929 16:31:47.495900 477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:31:47.498677 491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:47.500031 477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:31:47.500133 477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "254c7fd5f3a24340948db9c5fc410003"
format_stamp: "Formatted at 2026-09-29 16:31:47 on dist-test-slave-mlmz"
server_key: "88d1ea7626799d9afaaecb26b81649cb"
server_key_iv: "446d1f8e58bdbee458efbebf281a1397"
server_key_version: "encryptionkey@0"
I20260929 16:31:47.500240 477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:47.514665 477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:47.515229 477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:47.515411 477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:47.521944 477 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:45821
I20260929 16:31:47.521952 543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:45821 every 8 connection(s)
I20260929 16:31:47.522790 477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:47.525570 544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:47.526289 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 477
I20260929 16:31:47.526396 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:47.526690 23728 external_mini_cluster.cc:1428] Setting key a2fbc05c0c53b7b0d084e10c923c63e1
I20260929 16:31:47.531868 544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003: Bootstrap starting.
I20260929 16:31:47.534195 544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:47.535022 544 log.cc:824] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:47.536365 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:47.528006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.23.44.62:45821)
0929 16:31:47.528646 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:47.528667 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:47.528712 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:31:47.529573 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:47.529604 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:47.530068 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:47.530406 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:47.530830 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.532203 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.533081 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.533748 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.533898 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:47.534665 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:47.534671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:47.534685 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:47.535152 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:47.535217 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:47.535242 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:47.535346 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:47.535485 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 16:31:47.535737 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":107,"threads_started":1}
I20260929 16:31:47.536901 544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003: No bootstrap required, opened a new log
I20260929 16:31:47.540175 544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254c7fd5f3a24340948db9c5fc410003" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45821 } }
I20260929 16:31:47.540393 544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:47.540477 544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254c7fd5f3a24340948db9c5fc410003, State: Initialized, Role: FOLLOWER
I20260929 16:31:47.540854 544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254c7fd5f3a24340948db9c5fc410003" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45821 } }
I20260929 16:31:47.540994 544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:47.541078 544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:47.541190 544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:47.542021 544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254c7fd5f3a24340948db9c5fc410003" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45821 } }
I20260929 16:31:47.542340 544 leader_election.cc:302] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 254c7fd5f3a24340948db9c5fc410003; no voters:
I20260929 16:31:47.542624 544 leader_election.cc:288] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:47.542733 549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:47.542979 549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [term 1 LEADER]: Becoming Leader. State: Replica: 254c7fd5f3a24340948db9c5fc410003, State: Running, Role: LEADER
I20260929 16:31:47.543326 549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254c7fd5f3a24340948db9c5fc410003" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45821 } }
I20260929 16:31:47.543757 544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:47.544847 550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "254c7fd5f3a24340948db9c5fc410003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254c7fd5f3a24340948db9c5fc410003" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45821 } } }
I20260929 16:31:47.544979 550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:47.544898 551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 254c7fd5f3a24340948db9c5fc410003. Latest consensus state: current_term: 1 leader_uuid: "254c7fd5f3a24340948db9c5fc410003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254c7fd5f3a24340948db9c5fc410003" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45821 } } }
I20260929 16:31:47.545086 551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:47.545398 558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:47.547811 558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:47.552474 558 catalog_manager.cc:1347] Generated new cluster ID: aef22936bee14d17b03cecf378a0397b
I20260929 16:31:47.552526 558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:47.567237 558 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:47.568015 558 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:47.576958 558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 254c7fd5f3a24340948db9c5fc410003: Generated new TSK 0
I20260929 16:31:47.577596 558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:31:47.581326 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:45821
--builtin_ntp_servers=127.23.44.20:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 16:31:47.685514 568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:47.685730 568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:47.685765 568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:47.689733 568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:47.689774 568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:47.689851 568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:47.694458 568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:45821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:47.695385 568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:47.696641 568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:47.705538 574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:47.705516 573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:47.705516 576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:47.705829 568 server_base.cc:1034] running on GCE node
I20260929 16:31:47.706357 568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:47.706980 568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:47.708135 568 hybrid_clock.cc:648] HybridClock initialized: now 1790699507708110 us; error 39 us; skew 500 ppm
I20260929 16:31:47.709925 568 webserver.cc:460] Webserver started at http://127.23.44.1:40989/ using document root <none> and password file <none>
I20260929 16:31:47.710520 568 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:47.710582 568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:47.710811 568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:47.712728 568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "ad5a0eff8a274e819b2c6e3aa6079635"
format_stamp: "Formatted at 2026-09-29 16:31:47 on dist-test-slave-mlmz"
server_key: "0d0498035e9d0bc1da8708edc41025ed"
server_key_iv: "aeb9ccb9e5d43113d263cb1c5cd7068e"
server_key_version: "encryptionkey@0"
I20260929 16:31:47.713315 568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "ad5a0eff8a274e819b2c6e3aa6079635"
format_stamp: "Formatted at 2026-09-29 16:31:47 on dist-test-slave-mlmz"
server_key: "0d0498035e9d0bc1da8708edc41025ed"
server_key_iv: "aeb9ccb9e5d43113d263cb1c5cd7068e"
server_key_version: "encryptionkey@0"
I20260929 16:31:47.716660 568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:31:47.719041 582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:47.720191 568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:47.720320 568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "ad5a0eff8a274e819b2c6e3aa6079635"
format_stamp: "Formatted at 2026-09-29 16:31:47 on dist-test-slave-mlmz"
server_key: "0d0498035e9d0bc1da8708edc41025ed"
server_key_iv: "aeb9ccb9e5d43113d263cb1c5cd7068e"
server_key_version: "encryptionkey@0"
I20260929 16:31:47.720428 568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:47.730942 568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:47.731529 568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:47.731709 568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:47.732218 568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:47.733057 568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:47.733125 568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:47.733191 568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:47.733239 568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:47.741883 568 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:38475
I20260929 16:31:47.741904 695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:38475 every 8 connection(s)
I20260929 16:31:47.742688 568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:47.746891 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 568
I20260929 16:31:47.746979 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:47.747282 23728 external_mini_cluster.cc:1428] Setting key 272eb22974b721ebf0ad22c7ee3a0fc7
I20260929 16:31:47.749416 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:45821
--builtin_ntp_servers=127.23.44.20:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 16:31:47.752159 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:47.744414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:35643 (local address 127.23.44.62:45821)
0929 16:31:47.744550 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:47.744554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:47.745656 (+ 1102us) server_negotiation.cc:407] Connection header received
0929 16:31:47.747213 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:47.747216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:47.747280 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:47.747363 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:47.748307 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.748959 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.750245 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.750474 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.750586 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:47.751662 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:47.751666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:47.751669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:47.751682 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:47.751707 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:47.751716 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:47.751797 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:47.751956 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 16:31:47.752014 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:31:47.752945 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:47.744803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:45821 (local address 127.23.44.1:35643)
0929 16:31:47.745523 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:47.745561 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:47.746977 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:47.747526 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:47.747532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:47.747647 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:47.748169 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:47.748182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.749088 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.749091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:47.750115 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:47.750125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.750618 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.750631 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:47.750801 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:47.751415 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:47.751416 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:47.751453 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:47.751510 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:47.751799 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:47.751810 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:47.751819 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:31:47.751952 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 16:31:47.752209 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
I20260929 16:31:47.754129 696 heartbeater.cc:347] Connected to a master server at 127.23.44.62:45821
I20260929 16:31:47.754359 696 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:47.754820 696 heartbeater.cc:511] Master 127.23.44.62:45821 requested a full tablet report, sending...
I20260929 16:31:47.756438 508 ts_manager.cc:194] Registered new tserver with Master: ad5a0eff8a274e819b2c6e3aa6079635 (127.23.44.1:38475)
I20260929 16:31:47.758343 508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.44.1:35643
W20260929 16:31:47.857718 699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:47.857944 699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:47.857977 699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:47.861801 699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:47.861841 699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:47.861891 699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:47.866503 699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:45821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:47.867398 699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:47.868634 699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:47.877580 707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:47.877599 704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:47.877599 705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:47.877902 699 server_base.cc:1034] running on GCE node
I20260929 16:31:47.878237 699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:47.878765 699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:47.879935 699 hybrid_clock.cc:648] HybridClock initialized: now 1790699507879904 us; error 38 us; skew 500 ppm
I20260929 16:31:47.881788 699 webserver.cc:460] Webserver started at http://127.23.44.2:44743/ using document root <none> and password file <none>
I20260929 16:31:47.882284 699 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:47.882340 699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:47.882503 699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:47.884373 699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "05ee98ae271e4616b9944f4de7a28e41"
format_stamp: "Formatted at 2026-09-29 16:31:47 on dist-test-slave-mlmz"
server_key: "57c562b725d25384052e54cbc5c5f499"
server_key_iv: "a25f3c6a728b63e0b8735319350fd876"
server_key_version: "encryptionkey@0"
I20260929 16:31:47.884860 699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "05ee98ae271e4616b9944f4de7a28e41"
format_stamp: "Formatted at 2026-09-29 16:31:47 on dist-test-slave-mlmz"
server_key: "57c562b725d25384052e54cbc5c5f499"
server_key_iv: "a25f3c6a728b63e0b8735319350fd876"
server_key_version: "encryptionkey@0"
I20260929 16:31:47.888232 699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:31:47.890594 713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:47.891819 699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:31:47.891961 699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "05ee98ae271e4616b9944f4de7a28e41"
format_stamp: "Formatted at 2026-09-29 16:31:47 on dist-test-slave-mlmz"
server_key: "57c562b725d25384052e54cbc5c5f499"
server_key_iv: "a25f3c6a728b63e0b8735319350fd876"
server_key_version: "encryptionkey@0"
I20260929 16:31:47.892076 699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:47.910265 699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:47.910916 699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:47.911103 699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:47.911662 699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:47.912566 699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:47.912637 699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:47.912703 699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:47.912756 699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:47.921460 699 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:35487
I20260929 16:31:47.921482 826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:35487 every 8 connection(s)
I20260929 16:31:47.922281 699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:47.925221 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 699
I20260929 16:31:47.925352 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:47.925660 23728 external_mini_cluster.cc:1428] Setting key 7def489d0ff879ae2f047ee1efefdeb3
I20260929 16:31:47.927639 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:45821
--builtin_ntp_servers=127.23.44.20:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 16:31:47.932178 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:47.924019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:41317 (local address 127.23.44.62:45821)
0929 16:31:47.924163 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:47.924167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:47.925157 (+ 990us) server_negotiation.cc:407] Connection header received
0929 16:31:47.926905 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:47.926908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:47.926970 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:47.927057 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:47.928259 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.928864 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.929998 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.930421 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.930537 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:47.931584 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:47.931590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:47.931596 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:47.931613 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:47.931641 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:47.931655 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:47.931737 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:47.931880 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 16:31:47.931951 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:31:47.932825 829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:47.924378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:45821 (local address 127.23.44.2:41317)
0929 16:31:47.924993 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:47.925048 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:47.926676 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:47.927234 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:47.927243 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:47.927341 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:47.928021 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:47.928043 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.929002 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.929005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:47.929899 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:47.929908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:47.930567 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:47.930570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:47.930718 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:47.931338 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:47.931339 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:47.931375 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:47.931432 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:47.931751 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:47.931757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:47.931761 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:31:47.931853 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:31:47.932047 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260929 16:31:47.934032 827 heartbeater.cc:347] Connected to a master server at 127.23.44.62:45821
I20260929 16:31:47.934247 827 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:47.934734 827 heartbeater.cc:511] Master 127.23.44.62:45821 requested a full tablet report, sending...
I20260929 16:31:47.935703 508 ts_manager.cc:194] Registered new tserver with Master: 05ee98ae271e4616b9944f4de7a28e41 (127.23.44.2:35487)
I20260929 16:31:47.936262 508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.44.2:41317
W20260929 16:31:48.033978 830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:48.034198 830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:48.034233 830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:48.038323 830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:48.038371 830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:48.038456 830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:48.043056 830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:45821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:48.044037 830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:48.045503 830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:48.054741 838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:48.054798 830 server_base.cc:1034] running on GCE node
W20260929 16:31:48.054733 835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:48.054747 836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:48.055341 830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:48.055961 830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:48.057123 830 hybrid_clock.cc:648] HybridClock initialized: now 1790699508057111 us; error 22 us; skew 500 ppm
I20260929 16:31:48.058907 830 webserver.cc:460] Webserver started at http://127.23.44.3:35119/ using document root <none> and password file <none>
I20260929 16:31:48.059479 830 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:48.059541 830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:48.059734 830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:48.061687 830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "73be6fb5091a4077949c80516d3847fd"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "89426627202a8b5dab7343aae4fb1472"
server_key_iv: "6b5f46f147660a28e9c9138c5630f730"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.062232 830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "73be6fb5091a4077949c80516d3847fd"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "89426627202a8b5dab7343aae4fb1472"
server_key_iv: "6b5f46f147660a28e9c9138c5630f730"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.065650 830 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:31:48.068023 844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.069216 830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:48.069378 830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "73be6fb5091a4077949c80516d3847fd"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "89426627202a8b5dab7343aae4fb1472"
server_key_iv: "6b5f46f147660a28e9c9138c5630f730"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.069497 830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:48.089771 830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:48.090422 830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:48.090602 830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:48.091154 830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:48.092041 830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:48.092113 830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.092183 830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:48.092227 830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.100835 830 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:39171
I20260929 16:31:48.100863 957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:39171 every 8 connection(s)
I20260929 16:31:48.101725 830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:48.103551 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 830
I20260929 16:31:48.103672 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:48.103953 23728 external_mini_cluster.cc:1428] Setting key a3684c0d0a00a17781596980ced13e58
I20260929 16:31:48.110894 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:48.103510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:44849 (local address 127.23.44.62:45821)
0929 16:31:48.103652 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:48.103657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:48.104818 (+ 1161us) server_negotiation.cc:407] Connection header received
0929 16:31:48.106435 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:48.106438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:48.106484 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:48.106548 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:48.107495 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:48.108016 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:48.109122 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:48.109407 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:48.109498 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:48.110498 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:48.110501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:48.110503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:48.110514 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:48.110532 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:48.110540 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:48.110600 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:48.110733 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 16:31:48.110781 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:31:48.111692 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:48.103925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:45821 (local address 127.23.44.3:44849)
0929 16:31:48.104667 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:48.104718 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:48.106223 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:48.106757 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:48.106764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:48.106854 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:48.107359 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:48.107371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:48.108131 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:48.108134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:48.109002 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:48.109011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:48.109552 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:48.109555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:48.109701 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:48.110263 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:48.110264 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:48.110300 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:48.110374 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:48.110615 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:48.110621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:48.110625 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:31:48.110728 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 16:31:48.110920 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
I20260929 16:31:48.112747 958 heartbeater.cc:347] Connected to a master server at 127.23.44.62:45821
I20260929 16:31:48.112957 958 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:48.113479 958 heartbeater.cc:511] Master 127.23.44.62:45821 requested a full tablet report, sending...
I20260929 16:31:48.114410 508 ts_manager.cc:194] Registered new tserver with Master: 73be6fb5091a4077949c80516d3847fd (127.23.44.3:39171)
I20260929 16:31:48.114965 508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.44.3:44849
I20260929 16:31:48.117566 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:31:48.120187 547 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:48.119330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.23.44.62:45821)
0929 16:31:48.119470 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:48.119474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:48.119623 (+ 149us) server_negotiation.cc:407] Connection header received
0929 16:31:48.119760 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:48.119763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:48.119807 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:48.119889 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:48.120095 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40262: BlockingRecv error: recv got EOF from 127.0.0.1:40262 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:31:48.120496 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 568
I20260929 16:31:48.126151 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 699
I20260929 16:31:48.131100 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 830
I20260929 16:31:48.136044 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 477
2026-09-29T16:31:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (802 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T16:31:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:48Z Disabled control of system clock
I20260929 16:31:48.163800 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:46509
--webserver_interface=127.23.44.62
--webserver_port=0
--builtin_ntp_servers=127.23.44.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:46509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:31:48.268667 969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:48.268891 969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:48.268926 969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:48.272717 969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:48.272763 969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:48.272778 969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:48.272794 969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:48.272809 969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:48.277269 969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:46509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:46509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:48.278203 969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:48.279441 969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:48.287421 977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:48.287442 974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:48.287442 975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:48.287910 969 server_base.cc:1034] running on GCE node
I20260929 16:31:48.288444 969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:48.289508 969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:48.290690 969 hybrid_clock.cc:648] HybridClock initialized: now 1790699508290660 us; error 44 us; skew 500 ppm
I20260929 16:31:48.292446 969 webserver.cc:460] Webserver started at http://127.23.44.62:34961/ using document root <none> and password file <none>
I20260929 16:31:48.293017 969 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:48.293110 969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:48.293376 969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:48.295318 969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "45ea3d849b0c4275ab4064a9d71fa2f3"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "cbb043209e2b0dec345cb0ec4383f056"
server_key_iv: "d97366f49866b5403bb5fac02d37e6ab"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.295867 969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "45ea3d849b0c4275ab4064a9d71fa2f3"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "cbb043209e2b0dec345cb0ec4383f056"
server_key_iv: "d97366f49866b5403bb5fac02d37e6ab"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.299414 969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:31:48.301913 983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.303146 969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:48.303306 969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "45ea3d849b0c4275ab4064a9d71fa2f3"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "cbb043209e2b0dec345cb0ec4383f056"
server_key_iv: "d97366f49866b5403bb5fac02d37e6ab"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.303426 969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:48.308068 969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:48.308727 969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:48.308910 969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:48.315822 969 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:46509
I20260929 16:31:48.315846 1035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:46509 every 8 connection(s)
I20260929 16:31:48.316723 969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:48.318771 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 969
I20260929 16:31:48.318864 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:48.319144 23728 external_mini_cluster.cc:1428] Setting key e19a690ab40127c61e769ac669a9da7c
I20260929 16:31:48.320159 1036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:48.325714 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:48.320534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.23.44.62:46509)
0929 16:31:48.321110 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:48.321124 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:48.321157 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:31:48.321865 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:48.321881 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:48.322206 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:48.322468 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:48.324371 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:48.324375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:48.324386 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:48.324396 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:48.324451 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:48.324467 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:48.324617 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:48.324725 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 16:31:48.324997 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":124,"threads_started":1}
I20260929 16:31:48.326702 1036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3: Bootstrap starting.
I20260929 16:31:48.328763 1036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:48.329527 1036 log.cc:824] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:48.331372 1036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3: No bootstrap required, opened a new log
I20260929 16:31:48.334721 1036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ea3d849b0c4275ab4064a9d71fa2f3" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46509 } }
I20260929 16:31:48.334951 1036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:48.335032 1036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ea3d849b0c4275ab4064a9d71fa2f3, State: Initialized, Role: FOLLOWER
I20260929 16:31:48.335472 1036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ea3d849b0c4275ab4064a9d71fa2f3" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46509 } }
I20260929 16:31:48.335625 1036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:48.335711 1036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:48.335805 1036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:48.336643 1036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ea3d849b0c4275ab4064a9d71fa2f3" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46509 } }
I20260929 16:31:48.336958 1036 leader_election.cc:302] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45ea3d849b0c4275ab4064a9d71fa2f3; no voters:
I20260929 16:31:48.337239 1036 leader_election.cc:288] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:48.337402 1041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:48.337659 1041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 45ea3d849b0c4275ab4064a9d71fa2f3, State: Running, Role: LEADER
I20260929 16:31:48.337970 1041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ea3d849b0c4275ab4064a9d71fa2f3" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46509 } }
I20260929 16:31:48.338421 1036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:48.339546 1042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45ea3d849b0c4275ab4064a9d71fa2f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ea3d849b0c4275ab4064a9d71fa2f3" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46509 } } }
I20260929 16:31:48.339569 1043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45ea3d849b0c4275ab4064a9d71fa2f3. Latest consensus state: current_term: 1 leader_uuid: "45ea3d849b0c4275ab4064a9d71fa2f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ea3d849b0c4275ab4064a9d71fa2f3" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46509 } } }
I20260929 16:31:48.339668 1042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:48.339671 1043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:48.340118 1050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:48.343128 1050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:48.347980 1050 catalog_manager.cc:1347] Generated new cluster ID: 668b9ff2a35e44bea89c2bc15a4ccfe2
I20260929 16:31:48.348047 1050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:48.384011 1050 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:48.384327 1050 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:48.391011 1050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45ea3d849b0c4275ab4064a9d71fa2f3: Generated new TSK 0
I20260929 16:31:48.391630 1050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:31:48.397492 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:46509
--builtin_ntp_servers=127.23.44.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:31:48.510232 1060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:48.510501 1060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:48.510579 1060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:48.514585 1060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:48.514657 1060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:48.514750 1060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:48.519438 1060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:48.520465 1060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:48.521852 1060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:48.530951 1066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:48.530947 1065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:48.531020 1068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:48.531384 1060 server_base.cc:1034] running on GCE node
I20260929 16:31:48.531730 1060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:48.532264 1060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:48.533414 1060 hybrid_clock.cc:648] HybridClock initialized: now 1790699508533400 us; error 26 us; skew 500 ppm
I20260929 16:31:48.535156 1060 webserver.cc:460] Webserver started at http://127.23.44.1:36921/ using document root <none> and password file <none>
I20260929 16:31:48.535706 1060 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:48.535769 1060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:48.536016 1060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:48.538004 1060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "7339d8c70f0c4b2a96e621ff31bdef9e"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "a4e32cc66ce09984847578727deb2989"
server_key_iv: "e35db927d7c8c37137270972462b08ae"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.538520 1060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "7339d8c70f0c4b2a96e621ff31bdef9e"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "a4e32cc66ce09984847578727deb2989"
server_key_iv: "e35db927d7c8c37137270972462b08ae"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.542070 1060 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:31:48.544461 1074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.545658 1060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:48.545774 1060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "7339d8c70f0c4b2a96e621ff31bdef9e"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "a4e32cc66ce09984847578727deb2989"
server_key_iv: "e35db927d7c8c37137270972462b08ae"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.545874 1060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:48.552215 1060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:48.552934 1060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:48.553113 1060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:48.553782 1060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:48.554719 1060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:48.554765 1060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.554801 1060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:48.554854 1060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.563324 1060 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:35213
I20260929 16:31:48.563330 1187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:35213 every 8 connection(s)
I20260929 16:31:48.564160 1060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:48.569940 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:48.565890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:55773 (local address 127.23.44.62:46509)
0929 16:31:48.566040 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:48.566045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:48.567132 (+ 1087us) server_negotiation.cc:407] Connection header received
0929 16:31:48.568356 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:48.568359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:48.568393 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:48.568467 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:48.569570 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:48.569574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:48.569576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:48.569587 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:48.569605 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:48.569612 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:48.569689 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:48.569805 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 16:31:48.569840 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:31:48.570775 1190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:48.566302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:46509 (local address 127.23.44.1:55773)
0929 16:31:48.566988 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:48.567027 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:48.568181 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:48.568628 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:48.568634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:48.568718 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:48.569306 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:48.569311 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:48.569360 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:48.569428 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:48.569690 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:48.569696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:48.569700 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:31:48.569822 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 16:31:48.570047 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":117,"threads_started":1}
I20260929 16:31:48.571986 1188 heartbeater.cc:347] Connected to a master server at 127.23.44.62:46509
I20260929 16:31:48.572211 1188 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:48.572597 1188 heartbeater.cc:511] Master 127.23.44.62:46509 requested a full tablet report, sending...
I20260929 16:31:48.573655 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 1060
I20260929 16:31:48.573794 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:48.574069 23728 external_mini_cluster.cc:1428] Setting key 8ec906ec46cab3aeae5f525857c103a3
I20260929 16:31:48.574200 1000 ts_manager.cc:194] Registered new tserver with Master: 7339d8c70f0c4b2a96e621ff31bdef9e (127.23.44.1:35213)
I20260929 16:31:48.576273 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:46509
--builtin_ntp_servers=127.23.44.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:31:48.680753 1191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:48.681021 1191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:48.681092 1191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:48.685140 1191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:48.685220 1191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:48.685323 1191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:48.689971 1191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:48.690917 1191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:48.692188 1191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:48.701056 1197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:48.701045 1196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:48.701318 1191 server_base.cc:1034] running on GCE node
W20260929 16:31:48.701040 1199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:48.701798 1191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:48.702517 1191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:48.703711 1191 hybrid_clock.cc:648] HybridClock initialized: now 1790699508703697 us; error 24 us; skew 500 ppm
I20260929 16:31:48.705724 1191 webserver.cc:460] Webserver started at http://127.23.44.2:34055/ using document root <none> and password file <none>
I20260929 16:31:48.706326 1191 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:48.706431 1191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:48.706717 1191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:48.708990 1191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "ec5a42eab3af42ee80888923cffa75c3"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "3356d3997eb77e2605e1a11041a4c53d"
server_key_iv: "e9899c28f1ebf7df7b95323698d6d558"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.709622 1191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "ec5a42eab3af42ee80888923cffa75c3"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "3356d3997eb77e2605e1a11041a4c53d"
server_key_iv: "e9899c28f1ebf7df7b95323698d6d558"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.713335 1191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:31:48.715704 1205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.716848 1191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:48.716975 1191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "ec5a42eab3af42ee80888923cffa75c3"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "3356d3997eb77e2605e1a11041a4c53d"
server_key_iv: "e9899c28f1ebf7df7b95323698d6d558"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.717095 1191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:48.721383 1191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:48.721956 1191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:48.722138 1191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:48.722632 1191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:48.723448 1191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:48.723515 1191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.723573 1191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:48.723623 1191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.732106 1191 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:33401
I20260929 16:31:48.732121 1318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:33401 every 8 connection(s)
I20260929 16:31:48.732968 1191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:48.738631 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:48.734837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:39907 (local address 127.23.44.62:46509)
0929 16:31:48.735019 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:48.735023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:48.735770 (+ 747us) server_negotiation.cc:407] Connection header received
0929 16:31:48.736955 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:48.736958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:48.736992 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:48.737062 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:48.738145 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:48.738147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:48.738149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:48.738160 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:48.738179 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:48.738186 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:48.738258 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:48.738490 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 16:31:48.738525 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:31:48.739567 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:48.735007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:46509 (local address 127.23.44.2:39907)
0929 16:31:48.735649 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:48.735693 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:48.736787 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:48.737176 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:48.737181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:48.737420 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:48.737918 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:48.737920 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:48.737955 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:48.738014 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:48.738341 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:48.738350 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:48.738356 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:31:48.738529 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 16:31:48.738703 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260929 16:31:48.740640 1319 heartbeater.cc:347] Connected to a master server at 127.23.44.62:46509
I20260929 16:31:48.740851 1319 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:48.741219 1319 heartbeater.cc:511] Master 127.23.44.62:46509 requested a full tablet report, sending...
I20260929 16:31:48.741871 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 1191
I20260929 16:31:48.741997 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:48.742228 999 ts_manager.cc:194] Registered new tserver with Master: ec5a42eab3af42ee80888923cffa75c3 (127.23.44.2:33401)
I20260929 16:31:48.742255 23728 external_mini_cluster.cc:1428] Setting key 197cf9b3549d540c2fcb8b3a6b8eef17
I20260929 16:31:48.743955 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:46509
--builtin_ntp_servers=127.23.44.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:31:48.850584 1322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:48.850849 1322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:48.850912 1322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:48.854859 1322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:48.854935 1322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:48.855026 1322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:48.859606 1322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:48.860543 1322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:48.861845 1322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:48.870852 1328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:48.870852 1330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:48.870940 1322 server_base.cc:1034] running on GCE node
W20260929 16:31:48.870869 1327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:48.871490 1322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:48.872089 1322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:48.873312 1322 hybrid_clock.cc:648] HybridClock initialized: now 1790699508873278 us; error 44 us; skew 500 ppm
I20260929 16:31:48.875016 1322 webserver.cc:460] Webserver started at http://127.23.44.3:43121/ using document root <none> and password file <none>
I20260929 16:31:48.875551 1322 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:48.875638 1322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:48.875855 1322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:48.878234 1322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "59ae1743d3704180bb074bb107aca799"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "75dee11d54f72a3b120a120c17942470"
server_key_iv: "091e6b548250eae0e4e933ab324c316d"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.878788 1322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "59ae1743d3704180bb074bb107aca799"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "75dee11d54f72a3b120a120c17942470"
server_key_iv: "091e6b548250eae0e4e933ab324c316d"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.882373 1322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:31:48.884755 1336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.885933 1322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:48.886066 1322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "59ae1743d3704180bb074bb107aca799"
format_stamp: "Formatted at 2026-09-29 16:31:48 on dist-test-slave-mlmz"
server_key: "75dee11d54f72a3b120a120c17942470"
server_key_iv: "091e6b548250eae0e4e933ab324c316d"
server_key_version: "encryptionkey@0"
I20260929 16:31:48.886173 1322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:48.890538 1322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:48.891172 1322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:48.891345 1322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:48.891844 1322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:48.892762 1322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:48.892832 1322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.892915 1322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:48.892963 1322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:48.901470 1322 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:37097
I20260929 16:31:48.901508 1449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:37097 every 8 connection(s)
I20260929 16:31:48.902324 1322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:48.908270 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:48.904117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:47121 (local address 127.23.44.62:46509)
0929 16:31:48.904279 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:48.904283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:48.905188 (+ 905us) server_negotiation.cc:407] Connection header received
0929 16:31:48.906490 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:48.906493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:48.906537 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:48.906617 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:48.907762 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:48.907764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:48.907766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:31:48.907777 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:48.907796 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:31:48.907804 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:48.907924 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:48.908083 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 16:31:48.908120 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:31:48.909091 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:48.904450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:46509 (local address 127.23.44.3:47121)
0929 16:31:48.905051 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:48.905088 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:48.906281 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:48.906785 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:48.906792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:48.906874 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:48.907502 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:31:48.907504 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:48.907541 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:31:48.907608 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:48.907942 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:48.907950 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:48.907955 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:31:48.908101 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 16:31:48.908348 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260929 16:31:48.909528 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 1322
I20260929 16:31:48.909659 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:48.910013 23728 external_mini_cluster.cc:1428] Setting key 5ff4cb377edd0011382038263dbe0e5a
I20260929 16:31:48.910372 1450 heartbeater.cc:347] Connected to a master server at 127.23.44.62:46509
I20260929 16:31:48.910615 1450 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:48.911079 1450 heartbeater.cc:511] Master 127.23.44.62:46509 requested a full tablet report, sending...
I20260929 16:31:48.912106 999 ts_manager.cc:194] Registered new tserver with Master: 59ae1743d3704180bb074bb107aca799 (127.23.44.3:37097)
I20260929 16:31:48.923523 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:31:48.926235 1039 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:31:48.925425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.23.44.62:46509)
0929 16:31:48.925578 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:48.925581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:48.925706 (+ 125us) server_negotiation.cc:407] Connection header received
0929 16:31:48.925792 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:48.925795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:48.925851 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:48.925914 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:48.926097 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33158: BlockingRecv error: recv got EOF from 127.0.0.1:33158 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:31:48.926656 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 1060
I20260929 16:31:48.932217 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 1191
I20260929 16:31:48.937357 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 1322
I20260929 16:31:48.944018 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 969
2026-09-29T16:31:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (808 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:48 dist-test-slave-mlmz krb5kdc[1460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:48 dist-test-slave-mlmz krb5kdc[1460](info): set up 2 sockets
Sep 29 16:31:48 dist-test-slave-mlmz krb5kdc[1460](info): commencing operation
krb5kdc: starting...
W20260929 16:31:50.002722 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:31:50 dist-test-slave-mlmz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:31:50.124918 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:44107
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:44107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:50.231863 1473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:50.232098 1473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:50.232137 1473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:50.236080 1473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:50.236130 1473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:50.236150 1473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:50.236166 1473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:50.236227 1473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:50.241204 1473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:44107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:44107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:50.242136 1473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:50.243029 1473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:50.251698 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:50.251809 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:50.251866 1473 server_base.cc:1034] running on GCE node
W20260929 16:31:50.251785 1481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:50.252408 1473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:50.253527 1473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:50.254726 1473 hybrid_clock.cc:648] HybridClock initialized: now 1790699510254705 us; error 32 us; skew 500 ppm
Sep 29 16:31:50 dist-test-slave-mlmz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:50.262876 1473 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:31:50.263756 1473 webserver.cc:460] Webserver started at http://127.23.44.62:34819/ using document root <none> and password file <none>
I20260929 16:31:50.264339 1473 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:50.264436 1473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:50.264654 1473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:50.266708 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "3cd4937335224c60b460d1c6e69fd466"
format_stamp: "Formatted at 2026-09-29 16:31:50 on dist-test-slave-mlmz"
server_key: "4fc47e8bb37c00ba56c43ae4b9e7f822"
server_key_iv: "c93c94d006b3f66edbe6988678e8ebe8"
server_key_version: "encryptionkey@0"
I20260929 16:31:50.267267 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "3cd4937335224c60b460d1c6e69fd466"
format_stamp: "Formatted at 2026-09-29 16:31:50 on dist-test-slave-mlmz"
server_key: "4fc47e8bb37c00ba56c43ae4b9e7f822"
server_key_iv: "c93c94d006b3f66edbe6988678e8ebe8"
server_key_version: "encryptionkey@0"
I20260929 16:31:50.270948 1473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:31:50.273277 1488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:50.274464 1473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:31:50.274559 1473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "3cd4937335224c60b460d1c6e69fd466"
format_stamp: "Formatted at 2026-09-29 16:31:50 on dist-test-slave-mlmz"
server_key: "4fc47e8bb37c00ba56c43ae4b9e7f822"
server_key_iv: "c93c94d006b3f66edbe6988678e8ebe8"
server_key_version: "encryptionkey@0"
I20260929 16:31:50.274639 1473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:50.311643 1473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:50.314877 1473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:50.315033 1473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:50.321703 1473 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:44107
I20260929 16:31:50.321772 1540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:44107 every 8 connection(s)
I20260929 16:31:50.322631 1473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:50.325433 1541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:50.330610 1541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466: Bootstrap starting.
I20260929 16:31:50.330920 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 1473
I20260929 16:31:50.331034 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:31:50.331295 23728 external_mini_cluster.cc:1428] Setting key 65ee54a199562a907cee10ce93cdd208
I20260929 16:31:50.333065 1541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:50.333822 1541 log.cc:824] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:50.335845 1541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466: No bootstrap required, opened a new log
I20260929 16:31:50.339692 1541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd4937335224c60b460d1c6e69fd466" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44107 } }
I20260929 16:31:50.339919 1541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:50.340005 1541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd4937335224c60b460d1c6e69fd466, State: Initialized, Role: FOLLOWER
Sep 29 16:31:50 dist-test-slave-mlmz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:50.340483 1541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd4937335224c60b460d1c6e69fd466" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44107 } }
I20260929 16:31:50.340657 1541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:50.340729 1541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:50.340830 1541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:50.341848 1541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd4937335224c60b460d1c6e69fd466" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44107 } }
I20260929 16:31:50.342213 1541 leader_election.cc:302] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cd4937335224c60b460d1c6e69fd466; no voters:
I20260929 16:31:50.342546 1541 leader_election.cc:288] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:50.342602 1546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:50.342803 1546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [term 1 LEADER]: Becoming Leader. State: Replica: 3cd4937335224c60b460d1c6e69fd466, State: Running, Role: LEADER
I20260929 16:31:50.343137 1546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd4937335224c60b460d1c6e69fd466" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44107 } }
I20260929 16:31:50.343732 1541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:50.344249 1546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cd4937335224c60b460d1c6e69fd466. Latest consensus state: current_term: 1 leader_uuid: "3cd4937335224c60b460d1c6e69fd466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd4937335224c60b460d1c6e69fd466" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44107 } } }
I20260929 16:31:50.344442 1546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:50.344669 1547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cd4937335224c60b460d1c6e69fd466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd4937335224c60b460d1c6e69fd466" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44107 } } }
I20260929 16:31:50.344769 1547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:50.346657 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:50.332604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.23.44.62:44107)
0929 16:31:50.332988 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:50.333003 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:50.333036 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:31:50.333724 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:50.333741 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:50.334096 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:50.334351 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:50.335086 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:50.336525 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:50.337441 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:50.338130 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:50.338292 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:50.341053 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:50.341058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:50.341069 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:50.341093 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:50.343024 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:50.343529 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:50.343535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:50.343541 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:50.343660 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:50.343987 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:50.343990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:50.343991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:50.344534 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:50.345333 (+ 799us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:50.345492 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 16:31:50.345771 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":129,"threads_started":1}
W20260929 16:31:50.348284 1561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:31:50.348373 1561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:31:50.348486 1554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:50.349378 1554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:50.354362 1554 catalog_manager.cc:1347] Generated new cluster ID: 1a06eb1b8e744c0e9b6ed90b0e2f5e07
I20260929 16:31:50.354423 1554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:50.363500 1554 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:50.364617 1554 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:50.370340 1554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466: Generated new TSK 0
I20260929 16:31:50.371040 1554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:31:50.400997 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:44107
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:50.510157 1567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:50.510517 1567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:50.510581 1567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:50.515728 1567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:50.515786 1567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:50.515875 1567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:31:50.520984 1567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:50.521965 1567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:50.522929 1567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:50.532191 1575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:50.532298 1567 server_base.cc:1034] running on GCE node
W20260929 16:31:50.532204 1572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:50.532212 1573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:50.532829 1567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:50.533495 1567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:50.534685 1567 hybrid_clock.cc:648] HybridClock initialized: now 1790699510534670 us; error 24 us; skew 500 ppm
Sep 29 16:31:50 dist-test-slave-mlmz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:50.537591 1567 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:31:50.538334 1567 webserver.cc:460] Webserver started at http://127.23.44.1:40927/ using document root <none> and password file <none>
I20260929 16:31:50.538844 1567 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:50.538902 1567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:50.539058 1567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:50.540910 1567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "4b20b66916944e1ba0ba73008fd8dcd8"
format_stamp: "Formatted at 2026-09-29 16:31:50 on dist-test-slave-mlmz"
server_key: "e2c7b1a87673621661c724f2b4c0c870"
server_key_iv: "996809914ab9a9db3e96ce4869f9ddfc"
server_key_version: "encryptionkey@0"
I20260929 16:31:50.541473 1567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "4b20b66916944e1ba0ba73008fd8dcd8"
format_stamp: "Formatted at 2026-09-29 16:31:50 on dist-test-slave-mlmz"
server_key: "e2c7b1a87673621661c724f2b4c0c870"
server_key_iv: "996809914ab9a9db3e96ce4869f9ddfc"
server_key_version: "encryptionkey@0"
I20260929 16:31:50.545202 1567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 16:31:50.548033 1582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:50.549535 1567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:31:50.549669 1567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "4b20b66916944e1ba0ba73008fd8dcd8"
format_stamp: "Formatted at 2026-09-29 16:31:50 on dist-test-slave-mlmz"
server_key: "e2c7b1a87673621661c724f2b4c0c870"
server_key_iv: "996809914ab9a9db3e96ce4869f9ddfc"
server_key_version: "encryptionkey@0"
I20260929 16:31:50.549799 1567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:50.571722 1567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:50.574805 1567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:50.574995 1567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:50.575533 1567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:50.576301 1567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:50.576346 1567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:50.576406 1567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:50.576437 1567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:50.585670 1567 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:39569
I20260929 16:31:50.585825 1695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:39569 every 8 connection(s)
I20260929 16:31:50.586570 1567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:50.587337 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 1567
I20260929 16:31:50.587450 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:50.587718 23728 external_mini_cluster.cc:1428] Setting key c8ed9b825c59483c4bed0ed89eeae25a
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
Sep 29 16:31:50 dist-test-slave-mlmz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:50.607162 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:50.589612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:40659 (local address 127.23.44.62:44107)
0929 16:31:50.589775 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:50.589781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:50.590626 (+ 845us) server_negotiation.cc:407] Connection header received
0929 16:31:50.592587 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:50.592591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:50.592663 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:50.592754 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:50.594381 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:50.594903 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:50.596635 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:50.596852 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:50.596937 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:50.601998 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:50.602002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:50.602005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:50.602039 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:50.603978 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:50.605138 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:50.605142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:50.605144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:50.605234 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:50.605674 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:50.605677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:50.605678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:50.605852 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:50.605930 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:50.606795 (+ 865us) server_negotiation.cc:299] Negotiation successful
0929 16:31:50.606914 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:31:50.608076 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:50.589979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44107 (local address 127.23.44.1:40659)
0929 16:31:50.590459 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:50.590514 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:50.592359 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:50.592982 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:50.592994 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:50.593539 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:50.594214 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:50.594236 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:50.595033 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:50.595040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:50.596511 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:50.596524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:50.597012 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:50.597018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:50.597238 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:50.597860 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:50.597885 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:50.601805 (+ 3920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:50.604601 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:50.604610 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:50.604628 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:50.605011 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:50.605364 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:50.605371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:50.605374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:50.605573 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:50.605960 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:50.605969 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:50.606304 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 16:31:50.606552 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 16:31:50.606948 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":108,"threads_started":1}
I20260929 16:31:50.609629 1696 heartbeater.cc:347] Connected to a master server at 127.23.44.62:44107
I20260929 16:31:50.609890 1696 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:50.610360 1696 heartbeater.cc:511] Master 127.23.44.62:44107 requested a full tablet report, sending...
I20260929 16:31:50.611838 1505 ts_manager.cc:194] Registered new tserver with Master: 4b20b66916944e1ba0ba73008fd8dcd8 (127.23.44.1:39569)
I20260929 16:31:50.613030 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:40659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:31:50.618598 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:44107
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:50.734176 1701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:50.734424 1701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:50.734468 1701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:50.739109 1701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:50.739187 1701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:50.739290 1701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:31:50.749713 1701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:50.750972 1701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:50.752156 1701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:50.763134 1709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:50.763144 1707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:50.763144 1706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:50.763295 1701 server_base.cc:1034] running on GCE node
I20260929 16:31:50.763752 1701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:50.764384 1701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:50.765573 1701 hybrid_clock.cc:648] HybridClock initialized: now 1790699510765550 us; error 32 us; skew 500 ppm
Sep 29 16:31:50 dist-test-slave-mlmz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:50.768194 1701 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:31:50.768977 1701 webserver.cc:460] Webserver started at http://127.23.44.2:34851/ using document root <none> and password file <none>
I20260929 16:31:50.769582 1701 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:50.769668 1701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:50.769881 1701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:50.771770 1701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "51372b25d6004a8b841bd227853611b7"
format_stamp: "Formatted at 2026-09-29 16:31:50 on dist-test-slave-mlmz"
server_key: "d77debad085f9aa0164898d7dd8e1683"
server_key_iv: "135df552739c5be0300f52321113d6c0"
server_key_version: "encryptionkey@0"
I20260929 16:31:50.772310 1701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "51372b25d6004a8b841bd227853611b7"
format_stamp: "Formatted at 2026-09-29 16:31:50 on dist-test-slave-mlmz"
server_key: "d77debad085f9aa0164898d7dd8e1683"
server_key_iv: "135df552739c5be0300f52321113d6c0"
server_key_version: "encryptionkey@0"
I20260929 16:31:50.775964 1701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260929 16:31:50.778323 1716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:50.779563 1701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:31:50.779688 1701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "51372b25d6004a8b841bd227853611b7"
format_stamp: "Formatted at 2026-09-29 16:31:50 on dist-test-slave-mlmz"
server_key: "d77debad085f9aa0164898d7dd8e1683"
server_key_iv: "135df552739c5be0300f52321113d6c0"
server_key_version: "encryptionkey@0"
I20260929 16:31:50.779790 1701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:50.795473 1701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:50.798565 1701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:50.798756 1701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:50.799304 1701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:50.800069 1701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:50.800137 1701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:50.800207 1701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:50.800257 1701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:50.809394 1701 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:32773
I20260929 16:31:50.809466 1829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:32773 every 8 connection(s)
I20260929 16:31:50.810262 1701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:50.814790 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 1701
I20260929 16:31:50.814929 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:50.815207 23728 external_mini_cluster.cc:1428] Setting key fd57c1872275b08a3c62b2fdf7a43ca9
Sep 29 16:31:50 dist-test-slave-mlmz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:31:50.825105 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:50.812077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:42513 (local address 127.23.44.62:44107)
0929 16:31:50.812218 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:50.812227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:50.812952 (+ 725us) server_negotiation.cc:407] Connection header received
0929 16:31:50.814371 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:50.814375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:50.814433 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:50.814538 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:50.815873 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:50.816628 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:50.818396 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:50.818790 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:50.818945 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:50.821597 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:50.821606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:50.821608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:50.821632 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:50.823049 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:50.823577 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:50.823580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:50.823582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:50.823661 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:50.824001 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:50.824004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:50.824006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:50.824181 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:50.824288 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:50.824814 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 16:31:50.824927 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:31:50.825933 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:50.812402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44107 (local address 127.23.44.2:42513)
0929 16:31:50.812814 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:50.812852 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:50.814174 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:50.814746 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:50.814752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:50.815200 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:50.815688 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:50.815701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:50.816764 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:50.816768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:50.818231 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:50.818246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:50.818938 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:50.818941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:50.819077 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:50.819665 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:50.819688 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:50.821394 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:50.823188 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:50.823194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:50.823208 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:50.823462 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:50.823752 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:50.823756 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:50.823759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:50.823894 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:50.824284 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:50.824290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:50.824580 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 16:31:50.824755 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 16:31:50.825059 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
I20260929 16:31:50.827069 1830 heartbeater.cc:347] Connected to a master server at 127.23.44.62:44107
I20260929 16:31:50.827319 1830 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:50.827806 1830 heartbeater.cc:511] Master 127.23.44.62:44107 requested a full tablet report, sending...
I20260929 16:31:50.828826 1505 ts_manager.cc:194] Registered new tserver with Master: 51372b25d6004a8b841bd227853611b7 (127.23.44.2:32773)
I20260929 16:31:50.829717 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:42513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:31:50.840874 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:44107
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:42373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:50.946408 1835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:50.946632 1835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:50.946671 1835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:50.950524 1835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:50.950567 1835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:50.950619 1835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:31:50.956459 1835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:42373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:31:50.957353 1835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:50.958241 1835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:50.967769 1840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:50.967793 1841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:50.967795 1843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:50.967857 1835 server_base.cc:1034] running on GCE node
I20260929 16:31:50.968374 1835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:50.969048 1835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:50.970244 1835 hybrid_clock.cc:648] HybridClock initialized: now 1790699510970249 us; error 57 us; skew 500 ppm
Sep 29 16:31:50 dist-test-slave-mlmz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:50.972975 1835 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:31:50.973907 1835 webserver.cc:460] Webserver started at http://127.23.44.3:39711/ using document root <none> and password file <none>
I20260929 16:31:50.974447 1835 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:50.974534 1835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:50.974750 1835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:31:50.976689 1835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "31347c3282694937bafc82fe5fe6051c"
format_stamp: "Formatted at 2026-09-29 16:31:50 on dist-test-slave-mlmz"
server_key: "b6b3fbd2608f96a53e30f63bb774d16a"
server_key_iv: "1fb00e92143d5cb1cabe80665960d9ce"
server_key_version: "encryptionkey@0"
I20260929 16:31:50.977236 1835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "31347c3282694937bafc82fe5fe6051c"
format_stamp: "Formatted at 2026-09-29 16:31:50 on dist-test-slave-mlmz"
server_key: "b6b3fbd2608f96a53e30f63bb774d16a"
server_key_iv: "1fb00e92143d5cb1cabe80665960d9ce"
server_key_version: "encryptionkey@0"
I20260929 16:31:50.980979 1835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:31:50.983461 1850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:50.984658 1835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:31:50.984790 1835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "31347c3282694937bafc82fe5fe6051c"
format_stamp: "Formatted at 2026-09-29 16:31:50 on dist-test-slave-mlmz"
server_key: "b6b3fbd2608f96a53e30f63bb774d16a"
server_key_iv: "1fb00e92143d5cb1cabe80665960d9ce"
server_key_version: "encryptionkey@0"
I20260929 16:31:50.984897 1835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:50.993916 1835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:50.996587 1835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:50.996776 1835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:50.997385 1835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:50.998157 1835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:50.998227 1835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:50.998296 1835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:50.998350 1835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:51.006683 1835 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:35037
I20260929 16:31:51.006711 1963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:35037 every 8 connection(s)
I20260929 16:31:51.007481 1835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:51.016256 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 1835
I20260929 16:31:51.016387 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
Sep 29 16:31:51 dist-test-slave-mlmz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:31:51.016739 23728 external_mini_cluster.cc:1428] Setting key 9c99d1f84aa5bc8f141adc119d5efb40
I20260929 16:31:51.020759 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.009150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:47221 (local address 127.23.44.62:44107)
0929 16:31:51.009325 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:51.009330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:51.010065 (+ 735us) server_negotiation.cc:407] Connection header received
0929 16:31:51.011413 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:51.011416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:51.011463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:51.011537 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:51.012567 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.013091 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.014219 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.014439 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.014530 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.017165 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:51.017171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:51.017176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:51.017209 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:51.018802 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:51.019308 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:51.019312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:51.019313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:51.019395 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:51.019732 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:51.019736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:51.019738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:51.019912 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:51.020022 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:51.020484 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 16:31:51.020598 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:31:51.021509 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.009527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:44107 (local address 127.23.44.3:47221)
0929 16:31:51.009925 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:51.009962 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:51.011196 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:51.011689 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:51.011695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:51.012006 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:51.012423 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:51.012435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.013225 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.013228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:51.014086 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:51.014096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.014570 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.014573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:51.014701 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.015297 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:51.015312 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:51.016982 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:51.018935 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:51.018940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:51.018960 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:51.019202 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:51.019502 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:51.019505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:51.019507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:51.019638 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:51.020018 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:51.020023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:51.020245 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 16:31:51.020418 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 16:31:51.020709 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260929 16:31:51.022943 1964 heartbeater.cc:347] Connected to a master server at 127.23.44.62:44107
I20260929 16:31:51.023484 1964 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:51.024634 1964 heartbeater.cc:511] Master 127.23.44.62:44107 requested a full tablet report, sending...
I20260929 16:31:51.025832 1505 ts_manager.cc:194] Registered new tserver with Master: 31347c3282694937bafc82fe5fe6051c (127.23.44.3:35037)
I20260929 16:31:51.026384 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:47221
I20260929 16:31:51.030506 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:31:51.039696 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.032453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.23.44.62:44107)
0929 16:31:51.032568 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:51.032571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:51.032584 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:31:51.032719 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:51.032721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:51.032759 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:51.032829 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:51.033585 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.034069 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.034867 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.035077 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.035150 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.036940 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:51.036943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:51.036945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:51.036965 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:51.038302 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:51.038692 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:51.038694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:51.038695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:51.038767 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:51.038998 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:51.039000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:51.039001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:51.039186 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:51.039285 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:51.039498 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 16:31:51.039593 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:31:51.042623 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:51.045781 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:51.060262 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.054956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.23.44.1:39569)
0929 16:31:51.055615 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:51.055630 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:51.055651 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:31:51.055716 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:51.055721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:51.055905 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:51.056048 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:51.056309 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.057405 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.058732 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.059550 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.059688 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.059859 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:51.060039 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 16:31:51.060126 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":98,"threads_started":1}
I20260929 16:31:51.060258 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.054200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:39569 (local address 127.0.0.1:41226)
0929 16:31:51.055242 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:51.055257 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:51.055344 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:51.056082 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:51.056084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:51.056102 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:51.056195 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:51.056200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.057554 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.057558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:51.058572 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:51.058582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.059715 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.059721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:51.059822 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.059856 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:31:51.059987 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 16:31:51.060110 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260929 16:31:51.060745 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.054275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52584 (local address 127.23.44.3:35037)
0929 16:31:51.054922 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:51.054936 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:51.054958 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:31:51.055044 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:51.055053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:51.055321 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:51.055473 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:51.055933 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.057107 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.059101 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.059647 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.059748 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.059875 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:51.060324 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 16:31:51.060595 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":123,"threads_started":1}
I20260929 16:31:51.061002 1975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.054175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:35037 (local address 127.0.0.1:52584)
0929 16:31:51.054511 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:51.054555 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:51.054673 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:51.055656 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:51.055662 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:51.055687 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:51.055803 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:51.055810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.057244 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.057247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:51.058985 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:51.058994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.059934 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.059940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:51.060065 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.060095 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:31:51.060164 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:31:51.060229 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":45,"thread_start_us":117,"threads_started":1}
I20260929 16:31:51.062079 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.054892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:32773 (local address 127.0.0.1:50662)
0929 16:31:51.055710 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:51.055722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:51.055861 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:51.057309 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:51.057312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:51.057330 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:51.057420 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:51.057429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.058765 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.058768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:51.059900 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:51.059911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.061687 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.061691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:51.061781 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.061803 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:31:51.061886 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:31:51.061950 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260929 16:31:51.062690 1980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.055170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.23.44.2:32773)
0929 16:31:51.055561 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:51.055572 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:51.055792 (+ 220us) server_negotiation.cc:407] Connection header received
0929 16:31:51.055984 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:51.055991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:51.056165 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:51.056317 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:51.057565 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.058640 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.060031 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.061571 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.061680 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.061795 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:51.061916 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 16:31:51.062001 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":97,"threads_started":1}
I20260929 16:31:51.063419 1630 tablet_service.cc:1467] Processing CreateTablet for tablet 6a00612c24034371a629ab23a9b16b43 (DEFAULT_TABLE table=test-table [id=a4387ee884f5455eba68ececbc646d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:51.063869 1898 tablet_service.cc:1467] Processing CreateTablet for tablet 6a00612c24034371a629ab23a9b16b43 (DEFAULT_TABLE table=test-table [id=a4387ee884f5455eba68ececbc646d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:51.064507 1630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a00612c24034371a629ab23a9b16b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:51.064767 1898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a00612c24034371a629ab23a9b16b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:51.065135 1764 tablet_service.cc:1467] Processing CreateTablet for tablet 6a00612c24034371a629ab23a9b16b43 (DEFAULT_TABLE table=test-table [id=a4387ee884f5455eba68ececbc646d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:51.066059 1764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a00612c24034371a629ab23a9b16b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:51.070426 1981 tablet_bootstrap.cc:492] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c: Bootstrap starting.
I20260929 16:31:51.071794 1982 tablet_bootstrap.cc:492] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8: Bootstrap starting.
I20260929 16:31:51.072623 1981 tablet_bootstrap.cc:654] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:51.072623 1983 tablet_bootstrap.cc:492] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7: Bootstrap starting.
I20260929 16:31:51.073719 1982 tablet_bootstrap.cc:654] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:51.073941 1981 log.cc:824] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:51.074914 1982 log.cc:824] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:51.075095 1983 tablet_bootstrap.cc:654] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:51.075839 1983 log.cc:824] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:51.076117 1981 tablet_bootstrap.cc:492] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c: No bootstrap required, opened a new log
I20260929 16:31:51.076301 1981 ts_tablet_manager.cc:1397] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:31:51.076750 1982 tablet_bootstrap.cc:492] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8: No bootstrap required, opened a new log
I20260929 16:31:51.076923 1982 ts_tablet_manager.cc:1397] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:31:51.077703 1983 tablet_bootstrap.cc:492] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7: No bootstrap required, opened a new log
I20260929 16:31:51.077934 1983 ts_tablet_manager.cc:1397] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:31:51.080173 1981 raft_consensus.cc:348] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31347c3282694937bafc82fe5fe6051c" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 35037 } } peers { permanent_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 39569 } } peers { permanent_uuid: "51372b25d6004a8b841bd227853611b7" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32773 } }
I20260929 16:31:51.080435 1981 raft_consensus.cc:374] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:51.080490 1981 raft_consensus.cc:729] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31347c3282694937bafc82fe5fe6051c, State: Initialized, Role: FOLLOWER
I20260929 16:31:51.080547 1982 raft_consensus.cc:348] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31347c3282694937bafc82fe5fe6051c" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 35037 } } peers { permanent_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 39569 } } peers { permanent_uuid: "51372b25d6004a8b841bd227853611b7" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32773 } }
I20260929 16:31:51.080763 1982 raft_consensus.cc:374] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:51.080821 1982 raft_consensus.cc:729] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b20b66916944e1ba0ba73008fd8dcd8, State: Initialized, Role: FOLLOWER
I20260929 16:31:51.080914 1981 consensus_queue.cc:260] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31347c3282694937bafc82fe5fe6051c" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 35037 } } peers { permanent_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 39569 } } peers { permanent_uuid: "51372b25d6004a8b841bd227853611b7" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32773 } }
I20260929 16:31:51.081197 1982 consensus_queue.cc:260] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31347c3282694937bafc82fe5fe6051c" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 35037 } } peers { permanent_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 39569 } } peers { permanent_uuid: "51372b25d6004a8b841bd227853611b7" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32773 } }
I20260929 16:31:51.081183 1983 raft_consensus.cc:348] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31347c3282694937bafc82fe5fe6051c" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 35037 } } peers { permanent_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 39569 } } peers { permanent_uuid: "51372b25d6004a8b841bd227853611b7" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32773 } }
I20260929 16:31:51.081446 1983 raft_consensus.cc:374] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:51.081504 1983 raft_consensus.cc:729] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51372b25d6004a8b841bd227853611b7, State: Initialized, Role: FOLLOWER
I20260929 16:31:51.081768 1964 heartbeater.cc:503] Master 127.23.44.62:44107 was elected leader, sending a full tablet report...
I20260929 16:31:51.081861 1981 ts_tablet_manager.cc:1428] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 16:31:51.081933 1983 consensus_queue.cc:260] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31347c3282694937bafc82fe5fe6051c" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 35037 } } peers { permanent_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 39569 } } peers { permanent_uuid: "51372b25d6004a8b841bd227853611b7" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32773 } }
I20260929 16:31:51.082006 1982 ts_tablet_manager.cc:1428] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 16:31:51.082310 1696 heartbeater.cc:503] Master 127.23.44.62:44107 was elected leader, sending a full tablet report...
I20260929 16:31:51.082828 1983 ts_tablet_manager.cc:1428] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:31:51.083153 1830 heartbeater.cc:503] Master 127.23.44.62:44107 was elected leader, sending a full tablet report...
W20260929 16:31:51.087764 1697 tablet.cc:2369] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:31:51.125336 1988 raft_consensus.cc:482] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:51.125531 1988 raft_consensus.cc:504] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31347c3282694937bafc82fe5fe6051c" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 35037 } } peers { permanent_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 39569 } } peers { permanent_uuid: "51372b25d6004a8b841bd227853611b7" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32773 } }
I20260929 16:31:51.126592 1988 leader_election.cc:288] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31347c3282694937bafc82fe5fe6051c (127.23.44.3:35037), 51372b25d6004a8b841bd227853611b7 (127.23.44.2:32773)
I20260929 16:31:51.132457 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.126833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:35037 (local address 127.23.44.1:39149)
0929 16:31:51.127257 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:51.127269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:51.127450 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:51.127795 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:51.127798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:51.127842 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:51.128137 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:51.128161 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.129315 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.129319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:51.130863 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:51.130874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.131847 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.131851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:51.131991 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.132038 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 16:31:51.132103 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 16:31:51.132259 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":263,"spinlock_wait_cycles":7680,"thread_start_us":141,"threads_started":1}
I20260929 16:31:51.132460 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.126827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:32773 (local address 127.23.44.1:60899)
0929 16:31:51.127163 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:51.127178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:51.127306 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:51.127795 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:51.127798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:51.127841 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:51.128142 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:51.128147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.129315 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.129319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:51.130863 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:51.130874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.131847 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.131851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:51.131996 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.132021 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:31:51.132103 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:31:51.132259 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":346,"spinlock_wait_cycles":35840,"thread_start_us":94,"threads_started":1}
I20260929 16:31:51.132506 1980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.127029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:60899 (local address 127.23.44.2:32773)
0929 16:31:51.127178 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:51.127188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:51.127236 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:31:51.127600 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:51.127603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:51.127659 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:51.127749 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:51.128345 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.129170 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.131098 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.131722 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.131835 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.131904 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:51.132299 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 16:31:51.132377 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:31:51.132541 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.126963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:39149 (local address 127.23.44.3:35037)
0929 16:31:51.127093 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:51.127097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:51.127372 (+ 275us) server_negotiation.cc:407] Connection header received
0929 16:31:51.127600 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:51.127603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:51.127656 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:51.127744 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:51.128461 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.129170 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.131103 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.131722 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.131839 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.131904 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:51.132299 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 16:31:51.132377 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:31:51.133178 1918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00612c24034371a629ab23a9b16b43" candidate_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31347c3282694937bafc82fe5fe6051c" is_pre_election: true
I20260929 16:31:51.133373 1784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00612c24034371a629ab23a9b16b43" candidate_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51372b25d6004a8b841bd227853611b7" is_pre_election: true
I20260929 16:31:51.133584 1918 raft_consensus.cc:2454] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b20b66916944e1ba0ba73008fd8dcd8 in term 0.
I20260929 16:31:51.133692 1784 raft_consensus.cc:2454] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b20b66916944e1ba0ba73008fd8dcd8 in term 0.
I20260929 16:31:51.134044 1584 leader_election.cc:302] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31347c3282694937bafc82fe5fe6051c, 4b20b66916944e1ba0ba73008fd8dcd8; no voters:
I20260929 16:31:51.134337 1988 raft_consensus.cc:2781] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:51.134419 1988 raft_consensus.cc:482] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:51.134460 1988 raft_consensus.cc:3037] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:51.135437 1988 raft_consensus.cc:504] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31347c3282694937bafc82fe5fe6051c" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 35037 } } peers { permanent_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 39569 } } peers { permanent_uuid: "51372b25d6004a8b841bd227853611b7" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32773 } }
I20260929 16:31:51.135807 1988 leader_election.cc:288] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 31347c3282694937bafc82fe5fe6051c (127.23.44.3:35037), 51372b25d6004a8b841bd227853611b7 (127.23.44.2:32773)
I20260929 16:31:51.136121 1918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00612c24034371a629ab23a9b16b43" candidate_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31347c3282694937bafc82fe5fe6051c"
I20260929 16:31:51.136240 1784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00612c24034371a629ab23a9b16b43" candidate_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51372b25d6004a8b841bd227853611b7"
I20260929 16:31:51.136240 1918 raft_consensus.cc:3037] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:51.136382 1784 raft_consensus.cc:3037] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:51.137326 1918 raft_consensus.cc:2454] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b20b66916944e1ba0ba73008fd8dcd8 in term 1.
I20260929 16:31:51.137529 1784 raft_consensus.cc:2454] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b20b66916944e1ba0ba73008fd8dcd8 in term 1.
I20260929 16:31:51.137620 1584 leader_election.cc:302] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31347c3282694937bafc82fe5fe6051c, 4b20b66916944e1ba0ba73008fd8dcd8; no voters:
I20260929 16:31:51.137804 1988 raft_consensus.cc:2781] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:51.138065 1988 raft_consensus.cc:686] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 1 LEADER]: Becoming Leader. State: Replica: 4b20b66916944e1ba0ba73008fd8dcd8, State: Running, Role: LEADER
I20260929 16:31:51.138397 1988 consensus_queue.cc:237] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31347c3282694937bafc82fe5fe6051c" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 35037 } } peers { permanent_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 39569 } } peers { permanent_uuid: "51372b25d6004a8b841bd227853611b7" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32773 } }
I20260929 16:31:51.141603 1503 catalog_manager.cc:5495] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b20b66916944e1ba0ba73008fd8dcd8 (127.23.44.1). New cstate: current_term: 1 leader_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31347c3282694937bafc82fe5fe6051c" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 35037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b20b66916944e1ba0ba73008fd8dcd8" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 39569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51372b25d6004a8b841bd227853611b7" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32773 } health_report { overall_health: UNKNOWN } } }
I20260929 16:31:51.161247 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.157789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.23.44.1:39569)
0929 16:31:51.157938 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:51.157942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:51.157956 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:31:51.158099 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:51.158101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:51.158151 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:51.158257 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:51.158502 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.159000 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.159869 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.160086 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.160192 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.160485 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:51.160863 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:51.160994 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:51.161055 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 16:31:51.161108 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:31:51.166697 1784 raft_consensus.cc:1261] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7 [term 1 FOLLOWER]: Refusing update from remote peer 4b20b66916944e1ba0ba73008fd8dcd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:51.166800 1918 raft_consensus.cc:1261] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c [term 1 FOLLOWER]: Refusing update from remote peer 4b20b66916944e1ba0ba73008fd8dcd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:51.167487 1992 consensus_queue.cc:1035] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51372b25d6004a8b841bd227853611b7" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:51.167692 1988 consensus_queue.cc:1035] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31347c3282694937bafc82fe5fe6051c" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 35037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:51.177407 1997 mvcc.cc:205] Tried to move back new op lower bound from 7334705197731868672 to 7334705197626032128. Current Snapshot: MvccSnapshot[applied={T|T < 7334705197731868672}]
I20260929 16:31:51.177747 1996 mvcc.cc:205] Tried to move back new op lower bound from 7334705197731868672 to 7334705197626032128. Current Snapshot: MvccSnapshot[applied={T|T < 7334705197731868672 or (T in {7334705197731868672})}]
I20260929 16:31:51.184926 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:51.185079 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:51.187433 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cd4937335224c60b460d1c6e69fd466: Sending DeleteTablet for 3 replicas of tablet 6a00612c24034371a629ab23a9b16b43
I20260929 16:31:51.188218 1898 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a00612c24034371a629ab23a9b16b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:51 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:52584
I20260929 16:31:51.188236 1764 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a00612c24034371a629ab23a9b16b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:51 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:50662
I20260929 16:31:51.188323 1630 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a00612c24034371a629ab23a9b16b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:51 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:41226
I20260929 16:31:51.188771 2004 tablet_replica.cc:330] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c: stopping tablet replica
I20260929 16:31:51.188912 2005 tablet_replica.cc:330] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8: stopping tablet replica
I20260929 16:31:51.189118 2005 raft_consensus.cc:2229] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:51.189117 2004 raft_consensus.cc:2229] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:51.189419 2004 raft_consensus.cc:2258] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:51.189472 2005 raft_consensus.cc:2258] T 6a00612c24034371a629ab23a9b16b43 P 4b20b66916944e1ba0ba73008fd8dcd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:51.189644 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 1567
I20260929 16:31:51.192013 2004 ts_tablet_manager.cc:1905] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:51.194193 2004 ts_tablet_manager.cc:1918] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:51.194307 2004 log.cc:1196] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/6a00612c24034371a629ab23a9b16b43
I20260929 16:31:51.194489 2006 tablet_replica.cc:330] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7: stopping tablet replica
I20260929 16:31:51.194669 2004 ts_tablet_manager.cc:1939] T 6a00612c24034371a629ab23a9b16b43 P 31347c3282694937bafc82fe5fe6051c: Deleting consensus metadata
I20260929 16:31:51.194806 2006 raft_consensus.cc:2229] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:51.195116 2006 raft_consensus.cc:2258] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 16:31:51.196152 1489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:39569 (error 108)
I20260929 16:31:51.196348 1490 catalog_manager.cc:4841] TS 31347c3282694937bafc82fe5fe6051c (127.23.44.3:35037): tablet 6a00612c24034371a629ab23a9b16b43 (table test-table [id=a4387ee884f5455eba68ececbc646d47]) successfully deleted
I20260929 16:31:51.196694 2006 ts_tablet_manager.cc:1905] T 6a00612c24034371a629ab23a9b16b43 P 51372b25d6004a8b841bd227853611b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:51.197412 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.197078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:39569 (local address 127.0.0.1:41246)
0929 16:31:51.197215 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:51.197312 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:39569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 16:31:51.197571 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 1701
W20260929 16:31:51.201889 1489 catalog_manager.cc:4584] TS 4b20b66916944e1ba0ba73008fd8dcd8 (127.23.44.1:39569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a00612c24034371a629ab23a9b16b43: Network error: Client connection negotiation failed: client connection to 127.23.44.1:39569: connect: Connection refused (error 111)
I20260929 16:31:51.203850 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 1835
I20260929 16:31:51.204811 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.204053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:32773 (local address 127.0.0.1:50664)
0929 16:31:51.204655 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:51.204717 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.2:32773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506}
I20260929 16:31:51.209656 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 1473
2026-09-29T16:31:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2266 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:51 dist-test-slave-mlmz krb5kdc[2010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:51 dist-test-slave-mlmz krb5kdc[2010](info): set up 2 sockets
Sep 29 16:31:51 dist-test-slave-mlmz krb5kdc[2010](info): commencing operation
krb5kdc: starting...
W20260929 16:31:52.238269 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:31:52 dist-test-slave-mlmz krb5kdc[2010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.44.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.44.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/rangeradmin_127.23.44.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/rangeradmin_127.23.44.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.44.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.44.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/rangerlookup_127.23.44.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/rangerlookup_127.23.44.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.44.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.44.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/HTTP_127.23.44.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/HTTP_127.23.44.20@KRBTEST.COM.keytab.
I20260929 16:31:52.402558 23728 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNcOjI6/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/postgres -l logfile start
2026-09-29 16:31:55.068 UTC [2040] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:31:55.068 UTC [2040] LOG: listening on IPv4 address "127.23.44.20", port 56275
2026-09-29 16:31:55.072 UTC [2040] LOG: listening on Unix socket "/tmp/.s.PGSQL.56275"
2026-09-29 16:31:55.108 UTC [2043] LOG: database system was shut down at 2026-09-29 16:31:53 UTC
2026-09-29 16:31:55.112 UTC [2040] LOG: database system is ready to accept connections
I20260929 16:31:56.062500 23728 mini_postgres.cc:96] Postgres bound to 56275
2026-09-29 16:31:56.068 UTC [2051] FATAL: database "slave" does not exist
127.23.44.20:56275 - accepting connections
I20260929 16:31:56.069643 23728 mini_ranger.cc:161] Starting Ranger...
I20260929 16:31:56.084925 23728 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:31:56.605985 23728 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:31:56.606083 23728 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-admin
2026-09-29 16:31:56,907 [I] DB FLAVOR :POSTGRES
2026-09-29 16:31:56,908 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:31:56,908 [I] Checking connection..
2026-09-29 16:31:56,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:31:57,143 [I] Checking connection passed.
2026-09-29 16:31:57,143 [I] --------- Verifying version history table ---------
2026-09-29 16:31:57,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:31:57,409 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:31:57,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:31:57,643 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:31:57,644 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:31:57,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:31:57.895 UTC [2115] WARNING: there is no transaction in progress
2026-09-29 16:31:57,901 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:31:57,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:31:58,128 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:31:58,129 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:31:58,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:31:58,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:31:58,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mlmz.c.gcp-upstream.internal','N') ;"
2026-09-29 16:31:58,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:31:59,142 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:31:59,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:31:59,401 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:31:59,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:31:59,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:31:59,883 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:31:59,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:32:00.882 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:00.900 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:00.918 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:00.935 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:00.953 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:00.975 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:01.044 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:01.052 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:01.068 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:01.080 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:01.092 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:01.107 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:01.117 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:01.124 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:01.132 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:01.140 UTC [2223] WARNING: there is no transaction in progress
2026-09-29 16:32:01,831 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:32:01,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:32:02,067 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:32:02,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:32:02,302 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:32:02,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:32:02,558 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:32:02,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:32:02,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:32:03,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mlmz.c.gcp-upstream.internal';"
2026-09-29 16:32:03,293 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:32:03,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:32:03,513 [I] DB_PATCHES have already been applied
I20260929 16:32:03.519088 23728 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNcOjI6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:32:03.519156 23728 mini_ranger.cc:193] Using host: 127.23.44.20
I20260929 16:32:03.522771 23728 mini_ranger.cc:227] Ranger admin URL: http://127.23.44.20:33119
Sep 29, 2026 4:32:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:32:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:32:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:32:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:32:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:32:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:32:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:32:04.694450 23728 mini_ranger.cc:160] Time spent starting Ranger: real 8.625s user 0.003s sys 0.007s
Sep 29, 2026 4:32:24 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:32:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 4:32:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:32:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:32:24 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:32:25 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:32:26.456269 23728 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.44.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.44.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/rangerkms_127.23.44.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/rangerkms_127.23.44.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.44.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.44.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.44.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/HTTP_127.23.44.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.44.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/HTTP_127.23.44.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:32:26 dist-test-slave-mlmz krb5kdc[2010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699546, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:32:26.541668 23728 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:32:26.559548 23728 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:32:27.414628 23728 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:32:27.414723 23728 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-kms
I20260929 16:32:27.414825 23728 mini_ranger_kms.cc:227] Using postgres at 127.23.44.20:56275
2026-09-29 16:32:27,586 [I] DB FLAVOR :POSTGRES
2026-09-29 16:32:27,587 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:32:27,587 [I] Checking connection
2026-09-29 16:32:27,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:32:27,826 [I] connection success
2026-09-29 16:32:27,826 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:32:27,826 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:32:27,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:32:28,047 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:32:28,047 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:32:28,047 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:32:28,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:56275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:32:28,288 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:32:28.755012 23728 mini_ranger_kms.cc:310] Created kms service
I20260929 16:32:28.953159 23728 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:32:28.994293 23728 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:32:28 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699546, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:32:29.368934 23728 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:32:29.369287 23728 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:32:29.369323 23728 mini_ranger_kms.cc:240] Using host: 127.23.44.20
I20260929 16:32:29.373546 23728 mini_ranger_kms.cc:276] Ranger KMS PID: 2403
I20260929 16:32:29.373665 23728 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.44.20:52149
Sep 29, 2026 4:32:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:32:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:32:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:32:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:32:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:32:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 16:32:31.624699 23728 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.083s user 0.020s sys 0.137s
I20260929 16:32:31.624991 23728 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:32:31.625245 23728 mini_ranger_kms.cc:401] 127.23.44.20:52149/kms/v1/keys
Sep 29 16:32:31 dist-test-slave-mlmz krb5kdc[2010](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699551, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.44.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 16:32:32 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699551, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.44.20@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
Sep 29 16:32:32 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699551, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.44.20@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
Sep 29 16:32:34 dist-test-slave-mlmz krb5kdc[2010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:32:34.589003 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:34129
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:34129
--ranger_config_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.44.20:52149/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:34.718853 2454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:34.719084 2454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:34.719121 2454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:34.723017 2454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:32:34.723064 2454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:34.723083 2454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:34.723099 2454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:32:34.723136 2454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:32:34.728505 2454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.44.20:52149/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:34129
--ranger_config_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:34129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:32:34.729465 2454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:34.730410 2454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:34.738865 2459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:34.739008 2454 server_base.cc:1034] running on GCE node
W20260929 16:32:34.738844 2460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:34.738876 2462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:34.739813 2454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:34.741039 2454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:34.742225 2454 hybrid_clock.cc:648] HybridClock initialized: now 1790699554742198 us; error 40 us; skew 500 ppm
Sep 29 16:32:34 dist-test-slave-mlmz krb5kdc[2010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699554, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:34.745438 2454 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:32:34.746464 2454 webserver.cc:460] Webserver started at http://127.23.44.62:33541/ using document root <none> and password file <none>
I20260929 16:32:34.747094 2454 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:34.747169 2454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:34.747421 2454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:32:34 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699554, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:32:34.807977 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "aaf9a7bc33234fc791449764b62b7b36"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-mlmz"
server_key: "0924a2946fb0c53dc5fc23f7ba56a2ff"
server_key_iv: "4695f4b63dc53f08788d8b1b501f1f5e"
server_key_version: "kuduclusterkey@0"
I20260929 16:32:34.808564 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "aaf9a7bc33234fc791449764b62b7b36"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-mlmz"
server_key: "0924a2946fb0c53dc5fc23f7ba56a2ff"
server_key_iv: "4695f4b63dc53f08788d8b1b501f1f5e"
server_key_version: "kuduclusterkey@0"
I20260929 16:32:34.812351 2454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:32:34.828203 2469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:34.829576 2454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:32:34.829705 2454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "aaf9a7bc33234fc791449764b62b7b36"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-mlmz"
server_key: "0924a2946fb0c53dc5fc23f7ba56a2ff"
server_key_iv: "4695f4b63dc53f08788d8b1b501f1f5e"
server_key_version: "kuduclusterkey@0"
I20260929 16:32:34.829823 2454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:34.839190 2454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:34.842027 2454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:34.842213 2454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:34.849869 2454 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:34129
I20260929 16:32:34.849913 2521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:34129 every 8 connection(s)
I20260929 16:32:34.850776 2454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:32:34.853430 2522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:34.856206 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 2454
I20260929 16:32:34.856333 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:32:34.856619 23728 external_mini_cluster.cc:1428] Setting key 230e88be459aef17efd609dd907c88d5
I20260929 16:32:34.859164 2522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36: Bootstrap starting.
I20260929 16:32:34.861644 2522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:34.862448 2522 log.cc:824] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:34.864454 2522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36: No bootstrap required, opened a new log
Sep 29 16:32:34 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:32:34.868664 2522 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf9a7bc33234fc791449764b62b7b36" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34129 } }
I20260929 16:32:34.868913 2522 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:34.869020 2522 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf9a7bc33234fc791449764b62b7b36, State: Initialized, Role: FOLLOWER
I20260929 16:32:34.869515 2522 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf9a7bc33234fc791449764b62b7b36" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34129 } }
I20260929 16:32:34.869756 2522 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:32:34.869832 2522 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:32:34.869971 2522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:34.871050 2522 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf9a7bc33234fc791449764b62b7b36" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34129 } }
I20260929 16:32:34.871420 2522 leader_election.cc:302] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaf9a7bc33234fc791449764b62b7b36; no voters:
I20260929 16:32:34.871744 2522 leader_election.cc:288] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:32:34.871848 2527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:32:34.872138 2527 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [term 1 LEADER]: Becoming Leader. State: Replica: aaf9a7bc33234fc791449764b62b7b36, State: Running, Role: LEADER
I20260929 16:32:34.872455 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:34.857971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.23.44.62:34129)
0929 16:32:34.858384 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:34.858402 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:34.858440 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:32:34.859205 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:34.859225 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:34.859614 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:34.859940 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:34.860745 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.862204 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.863281 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.863912 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.864062 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:34.867144 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:34.867151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:34.867166 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:34.867195 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:34.869144 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:34.869664 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:34.869670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:34.869675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:34.869801 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:34.870140 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:34.870144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:34.870146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:34.870606 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:34.870795 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:34.871091 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 16:32:34.871540 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260929 16:32:34.872493 2527 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf9a7bc33234fc791449764b62b7b36" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34129 } }
I20260929 16:32:34.872967 2522 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:32:34.873911 2527 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaf9a7bc33234fc791449764b62b7b36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf9a7bc33234fc791449764b62b7b36" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34129 } } }
I20260929 16:32:34.874037 2527 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:34.874239 2527 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaf9a7bc33234fc791449764b62b7b36. Latest consensus state: current_term: 1 leader_uuid: "aaf9a7bc33234fc791449764b62b7b36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf9a7bc33234fc791449764b62b7b36" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34129 } } }
I20260929 16:32:34.874329 2527 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:34.875160 2522 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:32:34.880746 2536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:32:34.884759 2536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:32:34.891500 2536 catalog_manager.cc:1347] Generated new cluster ID: aedef4848cc94905a9b8e1c24a5dd1d2
I20260929 16:32:34.891558 2536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:32:34.911890 2536 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:32:34.913185 2536 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:32:34.922834 2536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36: Generated new TSK 0
I20260929 16:32:34.923579 2536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:32:35.584820 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.44.20:52149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34129
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:35.697625 2567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:35.697839 2567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:35.697878 2567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:35.701864 2567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:35.701937 2567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:35.702039 2567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:32:35.707289 2567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.44.20:52149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:32:35.708277 2567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:35.709240 2567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:35.720846 2572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:35.721484 2567 server_base.cc:1034] running on GCE node
W20260929 16:32:35.722024 2573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:35.723404 2575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:35.723814 2567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:35.724442 2567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:35.725646 2567 hybrid_clock.cc:648] HybridClock initialized: now 1790699555725622 us; error 41 us; skew 500 ppm
Sep 29 16:32:35 dist-test-slave-mlmz krb5kdc[2010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699555, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:35.728583 2567 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:32:35.729619 2567 webserver.cc:460] Webserver started at http://127.23.44.1:42661/ using document root <none> and password file <none>
I20260929 16:32:35.730331 2567 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:35.730423 2567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:35.730652 2567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:32:35 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699555, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:32:35.749933 2567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7"
format_stamp: "Formatted at 2026-09-29 16:32:35 on dist-test-slave-mlmz"
server_key: "aceb4a70b22bab5259332eef34601c3e"
server_key_iv: "1e009191c64b239624193c69f8f4037e"
server_key_version: "kuduclusterkey@0"
I20260929 16:32:35.750560 2567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7"
format_stamp: "Formatted at 2026-09-29 16:32:35 on dist-test-slave-mlmz"
server_key: "aceb4a70b22bab5259332eef34601c3e"
server_key_iv: "1e009191c64b239624193c69f8f4037e"
server_key_version: "kuduclusterkey@0"
I20260929 16:32:35.754546 2567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:32:35.774121 2584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:35.775830 2567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:32:35.775946 2567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7"
format_stamp: "Formatted at 2026-09-29 16:32:35 on dist-test-slave-mlmz"
server_key: "aceb4a70b22bab5259332eef34601c3e"
server_key_iv: "1e009191c64b239624193c69f8f4037e"
server_key_version: "kuduclusterkey@0"
I20260929 16:32:35.776058 2567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:35.816475 2567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:35.820060 2567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:35.820256 2567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:35.820919 2567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:35.821998 2567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:35.822118 2567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:35.822247 2567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:35.822348 2567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:35.832825 2567 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:42471
I20260929 16:32:35.832895 2698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:42471 every 8 connection(s)
I20260929 16:32:35.834722 2567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:32:35.849520 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 2567
I20260929 16:32:35.849648 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:32:35.849963 23728 external_mini_cluster.cc:1428] Setting key 86c1605a98018178731904c51e4a3614
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
Sep 29 16:32:35 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699555, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:32:35.876840 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:35.836766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:41519 (local address 127.23.44.62:34129)
0929 16:32:35.837071 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:35.837083 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:35.861557 (+ 24474us) server_negotiation.cc:407] Connection header received
0929 16:32:35.863011 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:35.863015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:35.863069 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:35.863157 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:35.864738 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.865655 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.869080 (+ 3425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.869461 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.869551 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:35.872535 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:35.872539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:35.872542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:35.872567 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:35.874439 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:35.875096 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:35.875101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:35.875103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:35.875192 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:35.875598 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:35.875603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:35.875605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:35.875805 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:35.875876 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:35.876541 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 16:32:35.876663 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
I20260929 16:32:35.878597 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:35.837184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:34129 (local address 127.23.44.1:41519)
0929 16:32:35.861361 (+ 24177us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:35.861442 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:35.862782 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:35.863479 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:35.863489 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:35.863832 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:35.864438 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:35.864454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.865824 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.865827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:35.868925 (+ 3098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:35.868938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.869758 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.869761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:35.869984 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:35.870668 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:35.870688 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:35.872319 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:35.874644 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:35.874650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:35.874664 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:35.874975 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:35.875296 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:35.875301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:35.875304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:35.875464 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:35.875952 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:35.875958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:35.876255 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 16:32:35.877310 (+ 1055us) client_negotiation.cc:240] Negotiation successful
0929 16:32:35.877659 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12142,"thread_start_us":139,"threads_started":1}
I20260929 16:32:35.880080 2700 heartbeater.cc:347] Connected to a master server at 127.23.44.62:34129
I20260929 16:32:35.880300 2700 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:35.880842 2700 heartbeater.cc:511] Master 127.23.44.62:34129 requested a full tablet report, sending...
I20260929 16:32:35.882573 2486 ts_manager.cc:194] Registered new tserver with Master: c52c5a9ddbd146d09ac67ac8ed01b9a7 (127.23.44.1:42471)
I20260929 16:32:35.883960 2486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:41519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:32:35.901217 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.44.20:52149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34129
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:32:35 dist-test-slave-mlmz krb5kdc[2010](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699555, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:32:36.041913 2706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:36.042162 2706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:36.042234 2706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:36.047083 2706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:36.047165 2706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:36.047266 2706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:32:36.053225 2706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.44.20:52149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:32:36.054221 2706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:36.055160 2706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:36.064211 2713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:36.064213 2715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:36.064316 2712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:36.065050 2706 server_base.cc:1034] running on GCE node
I20260929 16:32:36.065490 2706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:36.066121 2706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:36.067348 2706 hybrid_clock.cc:648] HybridClock initialized: now 1790699556067325 us; error 43 us; skew 500 ppm
Sep 29 16:32:36 dist-test-slave-mlmz krb5kdc[2010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699556, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:36.070209 2706 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:32:36.071244 2706 webserver.cc:460] Webserver started at http://127.23.44.2:38025/ using document root <none> and password file <none>
I20260929 16:32:36.071916 2706 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:36.071978 2706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:36.072182 2706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:32:36 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699556, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:32:36.086712 2706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "52b9c64fd87e4746bb72549dede9eca1"
format_stamp: "Formatted at 2026-09-29 16:32:36 on dist-test-slave-mlmz"
server_key: "13372b025bc844ee4f6f890db50fb630"
server_key_iv: "3705f15a5d2e0740231f18a28e7f2f41"
server_key_version: "kuduclusterkey@0"
I20260929 16:32:36.087585 2706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "52b9c64fd87e4746bb72549dede9eca1"
format_stamp: "Formatted at 2026-09-29 16:32:36 on dist-test-slave-mlmz"
server_key: "13372b025bc844ee4f6f890db50fb630"
server_key_iv: "3705f15a5d2e0740231f18a28e7f2f41"
server_key_version: "kuduclusterkey@0"
I20260929 16:32:36.095404 2706 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260929 16:32:36.115547 2722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:36.117411 2706 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 16:32:36.117527 2706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "52b9c64fd87e4746bb72549dede9eca1"
format_stamp: "Formatted at 2026-09-29 16:32:36 on dist-test-slave-mlmz"
server_key: "13372b025bc844ee4f6f890db50fb630"
server_key_iv: "3705f15a5d2e0740231f18a28e7f2f41"
server_key_version: "kuduclusterkey@0"
I20260929 16:32:36.117647 2706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:36.130208 2706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:36.133059 2706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:36.133301 2706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:36.133891 2706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:36.134905 2706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:36.135032 2706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:36.135144 2706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:36.135219 2706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:36.147078 2706 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:32775
I20260929 16:32:36.147909 2706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:32:36.148775 2835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:32775 every 8 connection(s)
I20260929 16:32:36.152305 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 2706
I20260929 16:32:36.152448 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:32:36.152761 23728 external_mini_cluster.cc:1428] Setting key 391d012871e26ec46545a3279f259c1a
Sep 29 16:32:36 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699556, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:32:36.162855 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.149782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:58871 (local address 127.23.44.62:34129)
0929 16:32:36.149902 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:36.149907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:36.150743 (+ 836us) server_negotiation.cc:407] Connection header received
0929 16:32:36.151686 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:36.151689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:36.151744 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:36.151820 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:36.153117 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.153848 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.155031 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.155295 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.155409 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.158281 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:36.158288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:36.158295 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:36.158329 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:36.160267 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:36.160928 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:36.160932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:36.160933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:36.161026 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:36.161463 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:36.161467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:36.161471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:36.161704 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:36.161785 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:36.162476 (+ 691us) server_negotiation.cc:299] Negotiation successful
0929 16:32:36.162612 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:32:36.164283 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.150182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:34129 (local address 127.23.44.2:58871)
0929 16:32:36.150601 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:36.150649 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:36.151490 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:36.151981 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:36.151987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:36.152319 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:36.152977 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.152992 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.153998 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.154005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.154905 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.154914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.155437 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.155441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.155635 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.156266 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:36.156291 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:36.158093 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:36.160422 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:36.160429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:36.160446 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:36.160803 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:36.161145 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:36.161148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:36.161150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:36.161329 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:36.161869 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:36.161875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:36.162177 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 16:32:36.163216 (+ 1039us) client_negotiation.cc:240] Negotiation successful
0929 16:32:36.163544 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":122,"threads_started":1}
I20260929 16:32:36.165320 2836 heartbeater.cc:347] Connected to a master server at 127.23.44.62:34129
I20260929 16:32:36.165558 2836 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:36.166060 2836 heartbeater.cc:511] Master 127.23.44.62:34129 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
I20260929 16:32:36.168001 2486 ts_manager.cc:194] Registered new tserver with Master: 52b9c64fd87e4746bb72549dede9eca1 (127.23.44.2:32775)
I20260929 16:32:36.168540 2486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:58871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:32:36.183761 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.44.20:52149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34129
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:36.323724 2841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:36.324087 2841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:36.324177 2841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:36.329944 2841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:36.330044 2841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:36.330157 2841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:32:36.337366 2841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.44.20:52149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:32:36.338640 2841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:36.339835 2841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:36.355305 2849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:36.355410 2846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:36.355306 2847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:36.356807 2841 server_base.cc:1034] running on GCE node
I20260929 16:32:36.357194 2841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:36.357951 2841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 29 16:32:36 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699555, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:32:36.373935 2841 hybrid_clock.cc:648] HybridClock initialized: now 1790699556373856 us; error 79 us; skew 500 ppm
Sep 29 16:32:36 dist-test-slave-mlmz krb5kdc[2010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699556, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:36.377910 2841 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:32:36.378952 2841 webserver.cc:460] Webserver started at http://127.23.44.3:42433/ using document root <none> and password file <none>
I20260929 16:32:36.379760 2841 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:36.379892 2841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:36.380211 2841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:32:36 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699556, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:32:36.402179 2841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "1d2560eb79f84c03b53f92724858a31e"
format_stamp: "Formatted at 2026-09-29 16:32:36 on dist-test-slave-mlmz"
server_key: "89e47f1f38db18e387cf0e8432c9fd2f"
server_key_iv: "c1ea4648814bf428e57d6b57e999b504"
server_key_version: "kuduclusterkey@0"
I20260929 16:32:36.402843 2841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "1d2560eb79f84c03b53f92724858a31e"
format_stamp: "Formatted at 2026-09-29 16:32:36 on dist-test-slave-mlmz"
server_key: "89e47f1f38db18e387cf0e8432c9fd2f"
server_key_iv: "c1ea4648814bf428e57d6b57e999b504"
server_key_version: "kuduclusterkey@0"
I20260929 16:32:36.420780 2841 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.004s sys 0.000s
I20260929 16:32:36.461661 2856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:36.463367 2841 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.000s
I20260929 16:32:36.463557 2841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "1d2560eb79f84c03b53f92724858a31e"
format_stamp: "Formatted at 2026-09-29 16:32:36 on dist-test-slave-mlmz"
server_key: "89e47f1f38db18e387cf0e8432c9fd2f"
server_key_iv: "c1ea4648814bf428e57d6b57e999b504"
server_key_version: "kuduclusterkey@0"
I20260929 16:32:36.463768 2841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:36.488155 2841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:36.491671 2841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:36.491860 2841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:36.492542 2841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:36.493561 2841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:36.493631 2841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:36.493737 2841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:36.493791 2841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:36.503796 2841 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:38991
I20260929 16:32:36.503854 2969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:38991 every 8 connection(s)
I20260929 16:32:36.505107 2841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:32:36.511373 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 2841
I20260929 16:32:36.511494 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:32:36.511761 23728 external_mini_cluster.cc:1428] Setting key a3ce553512f132c9ade524ae18e3d705
Sep 29 16:32:36 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699556, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:32:36.523151 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.509409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:53025 (local address 127.23.44.62:34129)
0929 16:32:36.509544 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:36.509549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:36.509568 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:32:36.510090 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:36.510095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:36.510157 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:36.510261 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:36.512661 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.513536 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.515457 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.515817 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.515927 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.518875 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:36.518879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:36.518882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:36.518907 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:36.520747 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:36.521447 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:36.521451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:36.521453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:36.521533 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:36.521949 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:36.521952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:36.521954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:36.522170 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:36.522249 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:36.522868 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 16:32:36.522984 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:32:36.524641 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.508198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:34129 (local address 127.23.44.3:53025)
0929 16:32:36.508766 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:36.508820 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:36.509834 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:36.510461 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:36.510469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:36.510805 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:36.512486 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.512505 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.513840 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.513843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.515313 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.515325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.515985 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.515989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.516183 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.516937 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:36.516958 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:36.518687 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:36.520893 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:36.520898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:36.520913 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:36.521330 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:36.521640 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:36.521643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:36.521645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:36.521839 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:36.522313 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:36.522319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:36.522612 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 16:32:36.523357 (+ 745us) client_negotiation.cc:240] Negotiation successful
0929 16:32:36.523666 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":154,"threads_started":1}
I20260929 16:32:36.526204 2970 heartbeater.cc:347] Connected to a master server at 127.23.44.62:34129
I20260929 16:32:36.526432 2970 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:36.526892 2970 heartbeater.cc:511] Master 127.23.44.62:34129 requested a full tablet report, sending...
I20260929 16:32:36.527940 2486 ts_manager.cc:194] Registered new tserver with Master: 1d2560eb79f84c03b53f92724858a31e (127.23.44.3:38991)
I20260929 16:32:36.528641 2486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:53025
I20260929 16:32:36.538003 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:32:36.551671 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.540163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.23.44.62:34129)
0929 16:32:36.540265 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:36.540269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:36.540535 (+ 266us) server_negotiation.cc:407] Connection header received
0929 16:32:36.540680 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:36.540683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:36.540728 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:36.540807 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:36.541669 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.542474 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.543683 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.544095 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.544200 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.545916 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:36.545920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:36.545923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:36.545946 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:36.547602 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:36.549891 (+ 2289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:36.549894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:36.549919 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:36.550029 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:36.550412 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:36.550438 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:36.550440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:36.550707 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:36.550863 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:36.551308 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 16:32:36.551466 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 16:32:36.557886 2486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:32:36.562340 2486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:32:36.582077 2982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.573908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:32775 (local address 127.0.0.1:42804)
0929 16:32:36.575606 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:36.575619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:36.575713 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:36.576805 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:36.576808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:36.576823 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:36.576904 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.576909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.578361 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.578365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.580324 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.580334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.581639 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.581643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.581726 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.581747 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:32:36.581810 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 16:32:36.581926 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1643,"mutex_wait_us":27,"thread_start_us":101,"threads_started":1}
I20260929 16:32:36.582571 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.573895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:42471 (local address 127.0.0.1:57718)
0929 16:32:36.575350 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:36.575362 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:36.575440 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:36.576551 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:36.576557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:36.576584 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:36.576710 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.576717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.578205 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.578209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.580342 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.580355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.581534 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.581539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.581668 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.581712 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 16:32:36.581800 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:32:36.582435 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1389,"mutex_wait_us":1215,"thread_start_us":86,"threads_started":1}
I20260929 16:32:36.582608 2987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.574362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42804 (local address 127.23.44.2:32775)
0929 16:32:36.574665 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:36.574676 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:36.575708 (+ 1032us) server_negotiation.cc:407] Connection header received
0929 16:32:36.575833 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:36.575840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:36.576042 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:36.577026 (+ 984us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:36.577154 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.578236 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.580469 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.581303 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.581398 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.581528 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:36.582391 (+ 863us) server_negotiation.cc:299] Negotiation successful
0929 16:32:36.582479 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260929 16:32:36.584033 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.574035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.23.44.1:42471)
0929 16:32:36.575948 (+ 1913us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:36.575955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:36.575977 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:32:36.576060 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:36.576065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:36.576250 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:36.576388 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:36.576852 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.578069 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.580483 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.581394 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.581937 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.582083 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:36.583808 (+ 1725us) server_negotiation.cc:299] Negotiation successful
0929 16:32:36.583904 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1810,"thread_start_us":99,"threads_started":1}
I20260929 16:32:36.585785 2752 tablet_service.cc:1467] Processing CreateTablet for tablet dacd784170d648328f4a06c2e154dff2 (DEFAULT_TABLE table=test-table [id=bc90369d627f4dd8b1c9c92181a6715d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:36.586925 2752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacd784170d648328f4a06c2e154dff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:36.587034 2633 tablet_service.cc:1467] Processing CreateTablet for tablet dacd784170d648328f4a06c2e154dff2 (DEFAULT_TABLE table=test-table [id=bc90369d627f4dd8b1c9c92181a6715d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:36.587988 2633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacd784170d648328f4a06c2e154dff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:36.589280 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.572382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:38991 (local address 127.0.0.1:56940)
0929 16:32:36.574931 (+ 2549us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:36.574978 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:36.575120 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:36.584230 (+ 9110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:36.584233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:36.584253 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:36.584332 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.584338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.586591 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.586597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.587682 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.587693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.588894 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.588898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.588986 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.589015 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:32:36.589074 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 16:32:36.589139 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2380,"thread_start_us":110,"threads_started":1}
I20260929 16:32:36.590085 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.572469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.23.44.3:38991)
0929 16:32:36.579520 (+ 7051us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:36.579535 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:36.579558 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:32:36.579631 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:36.579636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:36.579834 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:36.585131 (+ 5297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:36.585299 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.586461 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.587957 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.588759 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.589668 (+ 909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.589804 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:36.589852 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 16:32:36.589938 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6944,"thread_start_us":91,"threads_started":1}
I20260929 16:32:36.592763 2904 tablet_service.cc:1467] Processing CreateTablet for tablet dacd784170d648328f4a06c2e154dff2 (DEFAULT_TABLE table=test-table [id=bc90369d627f4dd8b1c9c92181a6715d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:36.593822 2904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacd784170d648328f4a06c2e154dff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:36.601678 2989 tablet_bootstrap.cc:492] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1: Bootstrap starting.
I20260929 16:32:36.602679 2990 tablet_bootstrap.cc:492] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e: Bootstrap starting.
I20260929 16:32:36.603633 2988 tablet_bootstrap.cc:492] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7: Bootstrap starting.
I20260929 16:32:36.605455 2990 tablet_bootstrap.cc:654] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:36.605674 2988 tablet_bootstrap.cc:654] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:36.606396 2990 log.cc:824] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:36.606472 2988 log.cc:824] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:36.608589 2989 tablet_bootstrap.cc:654] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:36.609400 2989 log.cc:824] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:36.609911 2988 tablet_bootstrap.cc:492] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7: No bootstrap required, opened a new log
I20260929 16:32:36.611181 2990 tablet_bootstrap.cc:492] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e: No bootstrap required, opened a new log
I20260929 16:32:36.611302 2989 tablet_bootstrap.cc:492] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1: No bootstrap required, opened a new log
I20260929 16:32:36.611363 2990 ts_tablet_manager.cc:1397] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 16:32:36.611455 2989 ts_tablet_manager.cc:1397] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20260929 16:32:36.615511 2990 raft_consensus.cc:348] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2560eb79f84c03b53f92724858a31e" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38991 } } peers { permanent_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 42471 } } peers { permanent_uuid: "52b9c64fd87e4746bb72549dede9eca1" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32775 } }
I20260929 16:32:36.615898 2990 raft_consensus.cc:374] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:36.616019 2990 raft_consensus.cc:729] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2560eb79f84c03b53f92724858a31e, State: Initialized, Role: FOLLOWER
I20260929 16:32:36.616340 2989 raft_consensus.cc:348] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2560eb79f84c03b53f92724858a31e" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38991 } } peers { permanent_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 42471 } } peers { permanent_uuid: "52b9c64fd87e4746bb72549dede9eca1" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32775 } }
I20260929 16:32:36.616572 2989 raft_consensus.cc:374] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:36.616621 2989 raft_consensus.cc:729] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b9c64fd87e4746bb72549dede9eca1, State: Initialized, Role: FOLLOWER
I20260929 16:32:36.616735 2990 consensus_queue.cc:260] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2560eb79f84c03b53f92724858a31e" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38991 } } peers { permanent_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 42471 } } peers { permanent_uuid: "52b9c64fd87e4746bb72549dede9eca1" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32775 } }
I20260929 16:32:36.617002 2989 consensus_queue.cc:260] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2560eb79f84c03b53f92724858a31e" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38991 } } peers { permanent_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 42471 } } peers { permanent_uuid: "52b9c64fd87e4746bb72549dede9eca1" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32775 } }
I20260929 16:32:36.617887 2989 ts_tablet_manager.cc:1428] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260929 16:32:36.617893 2990 ts_tablet_manager.cc:1428] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:32:36.618274 2970 heartbeater.cc:503] Master 127.23.44.62:34129 was elected leader, sending a full tablet report...
I20260929 16:32:36.618530 2988 ts_tablet_manager.cc:1397] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260929 16:32:36.618898 2836 heartbeater.cc:503] Master 127.23.44.62:34129 was elected leader, sending a full tablet report...
I20260929 16:32:36.622741 2988 raft_consensus.cc:348] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2560eb79f84c03b53f92724858a31e" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38991 } } peers { permanent_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 42471 } } peers { permanent_uuid: "52b9c64fd87e4746bb72549dede9eca1" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32775 } }
I20260929 16:32:36.623003 2988 raft_consensus.cc:374] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:36.623075 2988 raft_consensus.cc:729] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52c5a9ddbd146d09ac67ac8ed01b9a7, State: Initialized, Role: FOLLOWER
I20260929 16:32:36.623471 2988 consensus_queue.cc:260] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2560eb79f84c03b53f92724858a31e" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38991 } } peers { permanent_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 42471 } } peers { permanent_uuid: "52b9c64fd87e4746bb72549dede9eca1" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32775 } }
I20260929 16:32:36.624416 2988 ts_tablet_manager.cc:1428] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:32:36.624748 2700 heartbeater.cc:503] Master 127.23.44.62:34129 was elected leader, sending a full tablet report...
I20260929 16:32:36.627625 2994 raft_consensus.cc:482] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:32:36.627766 2994 raft_consensus.cc:504] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2560eb79f84c03b53f92724858a31e" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38991 } } peers { permanent_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 42471 } } peers { permanent_uuid: "52b9c64fd87e4746bb72549dede9eca1" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32775 } }
I20260929 16:32:36.628804 2994 leader_election.cc:288] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c52c5a9ddbd146d09ac67ac8ed01b9a7 (127.23.44.1:42471), 52b9c64fd87e4746bb72549dede9eca1 (127.23.44.2:32775)
I20260929 16:32:36.633823 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.629017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:42471 (local address 127.23.44.3:38639)
0929 16:32:36.629202 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:36.629219 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:36.629336 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:36.629821 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:36.629824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:36.629840 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:36.629947 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.629956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.631052 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.631055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.632286 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.632296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.633390 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.633395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.633520 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.633544 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:32:36.633620 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:32:36.633696 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 16:32:36.634495 2987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.629812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:60149 (local address 127.23.44.2:32775)
0929 16:32:36.629918 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:36.629922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:36.629934 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:32:36.629975 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:36.629977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:36.630027 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:36.630090 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:36.631104 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.631786 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.633170 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.633800 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.633878 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.633933 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:36.634321 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 16:32:36.634389 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:32:36.634497 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.629202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:38639 (local address 127.23.44.1:42471)
0929 16:32:36.629556 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:36.629560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:36.629573 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:32:36.629611 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:36.629614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:36.629655 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:36.629720 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:36.630157 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.630889 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.632494 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.633120 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.633223 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.633306 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:36.634312 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0929 16:32:36.634390 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260929 16:32:36.634863 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.629209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:32775 (local address 127.23.44.3:60149)
0929 16:32:36.629545 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:36.629558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:36.629639 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:36.630836 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:36.630839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:36.630857 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:36.630965 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.630970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.631934 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.631937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.633035 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:36.633045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.634090 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.634095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:36.634189 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.634209 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 16:32:36.634655 (+ 446us) client_negotiation.cc:240] Negotiation successful
0929 16:32:36.634723 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260929 16:32:36.635015 2653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacd784170d648328f4a06c2e154dff2" candidate_uuid: "1d2560eb79f84c03b53f92724858a31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7" is_pre_election: true
I20260929 16:32:36.635262 2653 raft_consensus.cc:2454] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d2560eb79f84c03b53f92724858a31e in term 0.
I20260929 16:32:36.635370 2790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacd784170d648328f4a06c2e154dff2" candidate_uuid: "1d2560eb79f84c03b53f92724858a31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b9c64fd87e4746bb72549dede9eca1" is_pre_election: true
I20260929 16:32:36.635644 2790 raft_consensus.cc:2454] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d2560eb79f84c03b53f92724858a31e in term 0.
I20260929 16:32:36.636077 2859 leader_election.cc:302] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d2560eb79f84c03b53f92724858a31e, 52b9c64fd87e4746bb72549dede9eca1; no voters:
I20260929 16:32:36.636587 2996 raft_consensus.cc:2781] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:32:36.636680 2996 raft_consensus.cc:482] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:32:36.636734 2996 raft_consensus.cc:3037] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:36.637830 2996 raft_consensus.cc:504] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2560eb79f84c03b53f92724858a31e" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38991 } } peers { permanent_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 42471 } } peers { permanent_uuid: "52b9c64fd87e4746bb72549dede9eca1" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32775 } }
I20260929 16:32:36.638142 2996 leader_election.cc:288] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [CANDIDATE]: Term 1 election: Requested vote from peers c52c5a9ddbd146d09ac67ac8ed01b9a7 (127.23.44.1:42471), 52b9c64fd87e4746bb72549dede9eca1 (127.23.44.2:32775)
I20260929 16:32:36.638614 2790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacd784170d648328f4a06c2e154dff2" candidate_uuid: "1d2560eb79f84c03b53f92724858a31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b9c64fd87e4746bb72549dede9eca1"
I20260929 16:32:36.638777 2790 raft_consensus.cc:3037] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:36.639850 2790 raft_consensus.cc:2454] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d2560eb79f84c03b53f92724858a31e in term 1.
I20260929 16:32:36.640159 2859 leader_election.cc:302] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d2560eb79f84c03b53f92724858a31e, 52b9c64fd87e4746bb72549dede9eca1; no voters:
I20260929 16:32:36.640301 2653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacd784170d648328f4a06c2e154dff2" candidate_uuid: "1d2560eb79f84c03b53f92724858a31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7"
I20260929 16:32:36.640408 2653 raft_consensus.cc:3037] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:36.641391 2653 raft_consensus.cc:2454] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d2560eb79f84c03b53f92724858a31e in term 1.
I20260929 16:32:36.641602 2996 raft_consensus.cc:2781] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:32:36.641696 2996 raft_consensus.cc:686] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 1 LEADER]: Becoming Leader. State: Replica: 1d2560eb79f84c03b53f92724858a31e, State: Running, Role: LEADER
I20260929 16:32:36.641956 2996 consensus_queue.cc:237] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2560eb79f84c03b53f92724858a31e" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38991 } } peers { permanent_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 42471 } } peers { permanent_uuid: "52b9c64fd87e4746bb72549dede9eca1" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32775 } }
I20260929 16:32:36.644702 2486 catalog_manager.cc:5495] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d2560eb79f84c03b53f92724858a31e (127.23.44.3). New cstate: current_term: 1 leader_uuid: "1d2560eb79f84c03b53f92724858a31e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2560eb79f84c03b53f92724858a31e" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 42471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52b9c64fd87e4746bb72549dede9eca1" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32775 } health_report { overall_health: UNKNOWN } } }
W20260929 16:32:36.651535 2837 tablet.cc:2369] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:32:36.662987 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.658699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.23.44.3:38991)
0929 16:32:36.658809 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:36.658816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:36.658829 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:32:36.659020 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:36.659024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:36.659083 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:36.659183 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:32:36.659699 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.660434 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.661616 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:36.661844 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:36.661953 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:36.662243 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:32:36.662600 (+ 357us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:32:36.662725 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:36.662789 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 16:32:36.662840 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:32:36.669445 2653 raft_consensus.cc:1261] T dacd784170d648328f4a06c2e154dff2 P c52c5a9ddbd146d09ac67ac8ed01b9a7 [term 1 FOLLOWER]: Refusing update from remote peer 1d2560eb79f84c03b53f92724858a31e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:32:36.670281 2994 consensus_queue.cc:1035] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c52c5a9ddbd146d09ac67ac8ed01b9a7" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 42471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:32:36.671434 2790 raft_consensus.cc:1261] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1 [term 1 FOLLOWER]: Refusing update from remote peer 1d2560eb79f84c03b53f92724858a31e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:32:36.671985 2994 consensus_queue.cc:1035] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b9c64fd87e4746bb72549dede9eca1" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:32:36.684595 3002 mvcc.cc:205] Tried to move back new op lower bound from 7334705384108638208 to 7334705384007688192. Current Snapshot: MvccSnapshot[applied={T|T < 7334705384108638208}]
I20260929 16:32:36.687228 3001 mvcc.cc:205] Tried to move back new op lower bound from 7334705384108638208 to 7334705384007688192. Current Snapshot: MvccSnapshot[applied={T|T < 7334705384108638208 or (T in {7334705384108638208})}]
I20260929 16:32:36.696630 2486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:32:36.696864 2486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:32:36.699478 2486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aaf9a7bc33234fc791449764b62b7b36: Sending DeleteTablet for 3 replicas of tablet dacd784170d648328f4a06c2e154dff2
I20260929 16:32:36.700132 2904 tablet_service.cc:1514] Processing DeleteTablet for tablet dacd784170d648328f4a06c2e154dff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:36 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:56940
I20260929 16:32:36.706924 2633 tablet_service.cc:1514] Processing DeleteTablet for tablet dacd784170d648328f4a06c2e154dff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:36 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:57718
I20260929 16:32:36.707819 2752 tablet_service.cc:1514] Processing DeleteTablet for tablet dacd784170d648328f4a06c2e154dff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:36 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:42804
I20260929 16:32:36.709141 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 2567
Sep 29 16:32:36 dist-test-slave-mlmz krb5kdc[2010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699555, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:32:36.716725 3015 tablet_replica.cc:330] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1: stopping tablet replica
I20260929 16:32:36.717095 3015 raft_consensus.cc:2229] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:32:36.722857 3013 tablet_replica.cc:330] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e: stopping tablet replica
W20260929 16:32:36.722890 2473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:42471 (error 108)
I20260929 16:32:36.723194 3013 raft_consensus.cc:2229] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:32:36.723547 3013 raft_consensus.cc:2258] T dacd784170d648328f4a06c2e154dff2 P 1d2560eb79f84c03b53f92724858a31e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:36.723806 3015 raft_consensus.cc:2258] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:36.725450 3015 ts_tablet_manager.cc:1905] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:32:36.728173 3015 ts_tablet_manager.cc:1918] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:32:36.728312 3015 log.cc:1196] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/dacd784170d648328f4a06c2e154dff2
I20260929 16:32:36.728677 3015 ts_tablet_manager.cc:1939] T dacd784170d648328f4a06c2e154dff2 P 52b9c64fd87e4746bb72549dede9eca1: Deleting consensus metadata
I20260929 16:32:36.730078 2472 catalog_manager.cc:4841] TS 52b9c64fd87e4746bb72549dede9eca1 (127.23.44.2:32775): tablet dacd784170d648328f4a06c2e154dff2 (table test-table [id=bc90369d627f4dd8b1c9c92181a6715d]) successfully deleted
I20260929 16:32:36.730511 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 2706
I20260929 16:32:36.737995 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:36.737623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:42471 (local address 127.0.0.1:57730)
0929 16:32:36.737829 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:36.737903 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260929 16:32:36.739603 2473 catalog_manager.cc:4584] TS c52c5a9ddbd146d09ac67ac8ed01b9a7 (127.23.44.1:42471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dacd784170d648328f4a06c2e154dff2: Network error: Client connection negotiation failed: client connection to 127.23.44.1:42471: connect: Connection refused (error 111)
I20260929 16:32:36.740487 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 2841
I20260929 16:32:36.748817 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 2454
I20260929 16:32:36.755070 23728 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:32:37.099232 23728 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:32:37.099370 23728 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:32:37.443737 23728 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:32:37.443 UTC [2040] LOG: received smart shutdown request
2026-09-29 16:32:37.447 UTC [2040] LOG: background worker "logical replication launcher" (PID 2049) exited with exit code 1
2026-09-29 16:32:37.447 UTC [2044] LOG: shutting down
2026-09-29 16:32:37.604 UTC [2040] LOG: database system is shut down
2026-09-29T16:32:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46506 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:32:37 dist-test-slave-mlmz krb5kdc[3026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:32:37 dist-test-slave-mlmz krb5kdc[3026](info): set up 2 sockets
Sep 29 16:32:37 dist-test-slave-mlmz krb5kdc[3026](info): commencing operation
krb5kdc: starting...
W20260929 16:32:38.757236 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:32:38 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:32:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.44.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.44.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/rangeradmin_127.23.44.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/rangeradmin_127.23.44.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.44.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.44.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/rangerlookup_127.23.44.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/rangerlookup_127.23.44.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.44.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.44.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/HTTP_127.23.44.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/HTTP_127.23.44.20@KRBTEST.COM.keytab.
I20260929 16:32:38.915932 23728 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNcOjI6/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/postgres -l logfile start
2026-09-29 16:32:41.332 UTC [3056] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:32:41.332 UTC [3056] LOG: listening on IPv4 address "127.23.44.20", port 60631
2026-09-29 16:32:41.335 UTC [3056] LOG: listening on Unix socket "/tmp/.s.PGSQL.60631"
2026-09-29 16:32:41.363 UTC [3059] LOG: database system was shut down at 2026-09-29 16:32:39 UTC
2026-09-29 16:32:41.366 UTC [3056] LOG: database system is ready to accept connections
I20260929 16:32:42.317133 23728 mini_postgres.cc:96] Postgres bound to 60631
2026-09-29 16:32:42.323 UTC [3067] FATAL: database "slave" does not exist
127.23.44.20:60631 - accepting connections
I20260929 16:32:42.323941 23728 mini_ranger.cc:161] Starting Ranger...
I20260929 16:32:42.338250 23728 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:32:42.800812 23728 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:32:42.800879 23728 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-admin
2026-09-29 16:32:43,063 [I] DB FLAVOR :POSTGRES
2026-09-29 16:32:43,064 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:32:43,064 [I] Checking connection..
2026-09-29 16:32:43,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:32:43,316 [I] Checking connection passed.
2026-09-29 16:32:43,316 [I] --------- Verifying version history table ---------
2026-09-29 16:32:43,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:32:43,566 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:32:43,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:32:43,790 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:32:43,790 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:32:43,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:32:44.039 UTC [3131] WARNING: there is no transaction in progress
2026-09-29 16:32:44,045 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:32:44,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:32:44,298 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:32:44,299 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:32:44,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:32:44,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:32:44,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mlmz.c.gcp-upstream.internal','N') ;"
2026-09-29 16:32:45,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:32:45,308 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:32:45,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:32:45,542 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:32:45,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:32:45,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:32:46,013 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:32:46,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:32:46.961 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:46.977 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:46.992 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.007 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.025 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.042 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.094 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.100 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.113 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.123 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.133 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.145 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.154 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.160 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.167 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47.174 UTC [3239] WARNING: there is no transaction in progress
2026-09-29 16:32:47,801 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:32:47,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:32:48,036 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:32:48,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:32:48,292 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:32:48,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:32:48,536 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:32:48,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:32:48,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:32:49,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mlmz.c.gcp-upstream.internal';"
2026-09-29 16:32:49,292 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:32:49,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:32:49,527 [I] DB_PATCHES have already been applied
I20260929 16:32:49.532388 23728 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNcOjI6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:32:49.532449 23728 mini_ranger.cc:193] Using host: 127.23.44.20
I20260929 16:32:49.535708 23728 mini_ranger.cc:227] Ranger admin URL: http://127.23.44.20:43021
Sep 29, 2026 4:32:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:32:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:32:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:32:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:32:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:32:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:32:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:32:50.646759 23728 mini_ranger.cc:160] Time spent starting Ranger: real 8.323s user 0.000s sys 0.009s
Sep 29, 2026 4:33:10 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:33:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 4:33:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:33:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:33:10 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:33:11 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:33:12.335969 23728 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.44.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.44.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/rangerkms_127.23.44.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.44.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/rangerkms_127.23.44.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.44.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.44.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.44.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/HTTP_127.23.44.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.44.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/HTTP_127.23.44.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:33:12 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699592, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:33:12.419062 23728 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:33:12.428409 23728 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:33:13.298172 23728 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:33:13.298259 23728 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-kms
I20260929 16:33:13.298277 23728 mini_ranger_kms.cc:227] Using postgres at 127.23.44.20:60631
2026-09-29 16:33:13,470 [I] DB FLAVOR :POSTGRES
2026-09-29 16:33:13,470 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:33:13,470 [I] Checking connection
2026-09-29 16:33:13,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:33:13,697 [I] connection success
2026-09-29 16:33:13,697 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:33:13,697 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:33:13,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:33:13,940 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:33:13,941 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:33:13,941 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:33:13,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.44.20:60631/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:33:14,186 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:33:14.664109 23728 mini_ranger_kms.cc:310] Created kms service
I20260929 16:33:14.842247 23728 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:33:14.896634 23728 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:33:14 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699592, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:33:15.288064 23728 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:33:15.288210 23728 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNcOjI6/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNcOjI6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:33:15.288232 23728 mini_ranger_kms.cc:240] Using host: 127.23.44.20
I20260929 16:33:15.292117 23728 mini_ranger_kms.cc:276] Ranger KMS PID: 3419
I20260929 16:33:15.292202 23728 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.44.20:50177
Sep 29, 2026 4:33:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:33:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:33:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:33:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:33:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:33:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 16:33:17 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699597, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.44.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:17.579422 23728 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.160s user 0.000s sys 0.153s
I20260929 16:33:17.579557 23728 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:33:17.579624 23728 mini_ranger_kms.cc:401] 127.23.44.20:50177/kms/v1/keys
Sep 29 16:33:17 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699597, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.44.20@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
Sep 29 16:33:18 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699597, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.44.20@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
Sep 29 16:33:20 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:33:20.481139 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:46671
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:46671
--ranger_config_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.44.20:50177/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:20.589951 3470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:20.590317 3470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:20.590427 3470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:20.594539 3470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:20.594587 3470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:20.594606 3470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:20.594622 3470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:20.594672 3470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:20.600646 3470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.44.20:50177/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:46671
--ranger_config_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:46671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:20.601613 3470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:20.602550 3470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:20.610555 3475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:20.610639 3470 server_base.cc:1034] running on GCE node
W20260929 16:33:20.610589 3478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:20.610646 3476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:20.611285 3470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:20.612258 3470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:20.613408 3470 hybrid_clock.cc:648] HybridClock initialized: now 1790699600613381 us; error 38 us; skew 500 ppm
Sep 29 16:33:20 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699600, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:20.621160 3470 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:33:20.622103 3470 webserver.cc:460] Webserver started at http://127.23.44.62:33201/ using document root <none> and password file <none>
I20260929 16:33:20.622586 3470 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:20.622635 3470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:20.622797 3470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:33:20 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699600, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:33:20.668957 3470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "57d735d8c879496c9e7b8e98e9a520f6"
format_stamp: "Formatted at 2026-09-29 16:33:20 on dist-test-slave-mlmz"
server_key: "e8fa8162a0a08f3ab82735f9f9016c3b"
server_key_iv: "5dda0b4dabbe28b975bb69a24b82c39a"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:20.669588 3470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "57d735d8c879496c9e7b8e98e9a520f6"
format_stamp: "Formatted at 2026-09-29 16:33:20 on dist-test-slave-mlmz"
server_key: "e8fa8162a0a08f3ab82735f9f9016c3b"
server_key_iv: "5dda0b4dabbe28b975bb69a24b82c39a"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:20.673297 3470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:33:20.690665 3485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:20.692173 3470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:33:20.692301 3470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "57d735d8c879496c9e7b8e98e9a520f6"
format_stamp: "Formatted at 2026-09-29 16:33:20 on dist-test-slave-mlmz"
server_key: "e8fa8162a0a08f3ab82735f9f9016c3b"
server_key_iv: "5dda0b4dabbe28b975bb69a24b82c39a"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:20.692420 3470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:20.712114 3470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:20.714685 3470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:20.714885 3470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:20.721823 3470 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:46671
I20260929 16:33:20.721848 3537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:46671 every 8 connection(s)
I20260929 16:33:20.722729 3470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:20.725894 3538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:20.727876 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 3470
I20260929 16:33:20.727998 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:33:20.728283 23728 external_mini_cluster.cc:1428] Setting key c2d0ab488a8aa510920d1fd3d32b4611
I20260929 16:33:20.731986 3538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6: Bootstrap starting.
I20260929 16:33:20.734515 3538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:20.735349 3538 log.cc:824] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:20.737097 3538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6: No bootstrap required, opened a new log
Sep 29 16:33:20 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:20.741060 3538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d735d8c879496c9e7b8e98e9a520f6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46671 } }
I20260929 16:33:20.741353 3538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:20.741463 3538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d735d8c879496c9e7b8e98e9a520f6, State: Initialized, Role: FOLLOWER
I20260929 16:33:20.741901 3538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d735d8c879496c9e7b8e98e9a520f6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46671 } }
I20260929 16:33:20.742024 3538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:20.742110 3538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:20.742202 3538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:20.742501 3541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:20.729640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.23.44.62:46671)
0929 16:33:20.730027 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:20.730046 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:20.730087 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:33:20.730961 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:20.730984 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:20.731373 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:20.731720 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:20.732373 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:20.733806 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:20.734659 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:20.735314 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:20.735480 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:20.737892 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:20.737898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:20.737914 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:20.737943 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:20.739697 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:20.740132 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:20.740136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:20.740141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:20.740236 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:20.740536 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:20.740538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:20.740539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:20.740866 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:20.741013 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:20.741369 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 16:33:20.741675 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
I20260929 16:33:20.743126 3538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d735d8c879496c9e7b8e98e9a520f6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46671 } }
I20260929 16:33:20.743475 3538 leader_election.cc:302] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57d735d8c879496c9e7b8e98e9a520f6; no voters:
I20260929 16:33:20.743757 3538 leader_election.cc:288] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:33:20.743849 3543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:20.744061 3543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 1 LEADER]: Becoming Leader. State: Replica: 57d735d8c879496c9e7b8e98e9a520f6, State: Running, Role: LEADER
I20260929 16:33:20.744385 3543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d735d8c879496c9e7b8e98e9a520f6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46671 } }
I20260929 16:33:20.745026 3538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:20.745560 3545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57d735d8c879496c9e7b8e98e9a520f6. Latest consensus state: current_term: 1 leader_uuid: "57d735d8c879496c9e7b8e98e9a520f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d735d8c879496c9e7b8e98e9a520f6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46671 } } }
I20260929 16:33:20.745759 3545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:20.745962 3544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57d735d8c879496c9e7b8e98e9a520f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d735d8c879496c9e7b8e98e9a520f6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46671 } } }
I20260929 16:33:20.746061 3544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:20.746239 3550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:20.748908 3550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:20.749311 3538 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:33:20.757390 3550 catalog_manager.cc:1347] Generated new cluster ID: eccdcfef201a458f8f0af3d713b59700
I20260929 16:33:20.757464 3550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:20.769774 3550 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:33:20.770969 3550 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:20.786921 3550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6: Generated new TSK 0
I20260929 16:33:20.787595 3550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:33:21.469046 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.44.20:50177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:46671
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:21.609371 3583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:21.609705 3583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:21.609786 3583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:21.615784 3583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:21.615859 3583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:21.615945 3583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:33:21.622460 3583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.44.20:50177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:21.623513 3583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:21.624650 3583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:21.636142 3591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:21.636178 3590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:21.636677 3593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:21.636699 3583 server_base.cc:1034] running on GCE node
I20260929 16:33:21.637341 3583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:21.638028 3583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:21.639190 3583 hybrid_clock.cc:648] HybridClock initialized: now 1790699601639171 us; error 35 us; skew 500 ppm
Sep 29 16:33:21 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699601, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:21.642186 3583 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:33:21.643074 3583 webserver.cc:460] Webserver started at http://127.23.44.1:35871/ using document root <none> and password file <none>
I20260929 16:33:21.643594 3583 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:21.643651 3583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:21.643843 3583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:33:21 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699601, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:33:21.664047 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "5c33f89b507a4ccd9093f8d2be20c9e1"
format_stamp: "Formatted at 2026-09-29 16:33:21 on dist-test-slave-mlmz"
server_key: "eb034099d11507faae7ec782af62bbd3"
server_key_iv: "35da608ab42febfa6f363fceca2e527c"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:21.664855 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "5c33f89b507a4ccd9093f8d2be20c9e1"
format_stamp: "Formatted at 2026-09-29 16:33:21 on dist-test-slave-mlmz"
server_key: "eb034099d11507faae7ec782af62bbd3"
server_key_iv: "35da608ab42febfa6f363fceca2e527c"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:21.669494 3583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 16:33:21.686105 3600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:21.687713 3583 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 16:33:21.687829 3583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "5c33f89b507a4ccd9093f8d2be20c9e1"
format_stamp: "Formatted at 2026-09-29 16:33:21 on dist-test-slave-mlmz"
server_key: "eb034099d11507faae7ec782af62bbd3"
server_key_iv: "35da608ab42febfa6f363fceca2e527c"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:21.687950 3583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:21.705605 3583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:21.708541 3583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:21.708729 3583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:21.709350 3583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:21.710193 3583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:21.710271 3583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:21.710378 3583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:21.710438 3583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:21.721547 3714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:33803 every 8 connection(s)
I20260929 16:33:21.721516 3583 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:33803
I20260929 16:33:21.722908 3583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:21.726665 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 3583
I20260929 16:33:21.726783 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:33:21.727186 23728 external_mini_cluster.cc:1428] Setting key c1296ab3fb3f2dd08454eda8854891f9
Sep 29 16:33:21 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699601, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:21.741708 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:21.725599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:46671 (local address 127.23.44.1:55441)
0929 16:33:21.726238 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:21.726286 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:21.727447 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:21.728102 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:21.728111 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:21.728449 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:21.729059 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:21.729082 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:21.730225 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:21.730228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:21.731526 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:21.731536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:21.732162 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:21.732165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:21.732344 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:21.733023 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:21.733047 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:21.734824 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:21.738587 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:21.738593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:21.738607 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:21.738924 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:21.739246 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:21.739249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:21.739251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:21.739408 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:21.739875 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:21.739882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:21.740193 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 16:33:21.740387 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 16:33:21.740694 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":118,"threads_started":1}
I20260929 16:33:21.742404 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:21.725460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:55441 (local address 127.23.44.62:46671)
0929 16:33:21.726002 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:21.726011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:21.726394 (+ 383us) server_negotiation.cc:407] Connection header received
0929 16:33:21.727673 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:21.727676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:21.727726 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:21.727811 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:21.729217 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:21.730085 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:21.731660 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:21.732013 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:21.732120 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:21.736555 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:21.736559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:21.736562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:21.736589 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:21.738456 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:21.739054 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:21.739058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:21.739060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:21.739149 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:21.739515 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:21.739518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:21.739520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:21.739731 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:21.739823 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:21.742135 (+ 2312us) server_negotiation.cc:299] Negotiation successful
0929 16:33:21.742254 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
I20260929 16:33:21.748639 3716 heartbeater.cc:347] Connected to a master server at 127.23.44.62:46671
I20260929 16:33:21.748874 3716 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:21.749464 3716 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
I20260929 16:33:21.751137 3502 ts_manager.cc:194] Registered new tserver with Master: 5c33f89b507a4ccd9093f8d2be20c9e1 (127.23.44.1:33803)
I20260929 16:33:21.752612 3502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:55441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:33:21.776475 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.44.20:50177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:46671
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:33:21 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699601, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:33:21.931552 3722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:21.931882 3722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:21.931957 3722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:21.938063 3722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:21.938136 3722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:21.938218 3722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:33:21.946113 3722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.44.20:50177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:21.947360 3722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:21.948637 3722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:33:21.961328 3722 server_base.cc:1034] running on GCE node
W20260929 16:33:21.961442 3728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:21.961639 3729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:21.961831 3731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:21.962366 3722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:21.963110 3722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:21.964315 3722 hybrid_clock.cc:648] HybridClock initialized: now 1790699601964295 us; error 33 us; skew 500 ppm
Sep 29 16:33:21 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699601, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:21.967886 3722 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:33:21.968891 3722 webserver.cc:460] Webserver started at http://127.23.44.2:44145/ using document root <none> and password file <none>
I20260929 16:33:21.969625 3722 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:21.969749 3722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:21.970018 3722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:33:21 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699601, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:33:21.991308 3722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "67f03a8864594abe96c07cf767ffab3b"
format_stamp: "Formatted at 2026-09-29 16:33:21 on dist-test-slave-mlmz"
server_key: "adc0f7fce6831c0a8555cee2d32c8504"
server_key_iv: "4c9a3b60b1130f110206c808304c78a8"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:21.992027 3722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "67f03a8864594abe96c07cf767ffab3b"
format_stamp: "Formatted at 2026-09-29 16:33:21 on dist-test-slave-mlmz"
server_key: "adc0f7fce6831c0a8555cee2d32c8504"
server_key_iv: "4c9a3b60b1130f110206c808304c78a8"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:21.997426 3722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:33:22.019065 3738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:22.020706 3722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:33:22.020821 3722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "67f03a8864594abe96c07cf767ffab3b"
format_stamp: "Formatted at 2026-09-29 16:33:21 on dist-test-slave-mlmz"
server_key: "adc0f7fce6831c0a8555cee2d32c8504"
server_key_iv: "4c9a3b60b1130f110206c808304c78a8"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:22.020936 3722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:22.035384 3722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:22.038939 3722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:22.039155 3722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:22.039768 3722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:22.040755 3722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:22.040859 3722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:22.040956 3722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:22.041029 3722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:22.055261 3722 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:33917
I20260929 16:33:22.056818 3722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:22.057955 3851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:33917 every 8 connection(s)
I20260929 16:33:22.063982 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 3722
I20260929 16:33:22.064114 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:33:22.064427 23728 external_mini_cluster.cc:1428] Setting key 87eaddd6cca93620af7fe4c8f906af2e
Sep 29 16:33:22 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699601, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:22.077342 3854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:22.059921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:46671 (local address 127.23.44.2:34605)
0929 16:33:22.061400 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:22.061452 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:22.062548 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:22.063290 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:22.063298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:22.063648 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:22.064254 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:22.064271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:22.066586 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:22.066589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:22.067993 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:22.068007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:22.068817 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:22.068821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:22.069019 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:22.069728 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:22.069748 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:22.071448 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:22.074098 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:22.074104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:22.074119 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:22.074464 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:22.074812 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:22.074816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:22.074819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:22.074986 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:22.075479 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:22.075485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:22.075810 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 16:33:22.076022 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 16:33:22.076334 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1258,"thread_start_us":138,"threads_started":1}
I20260929 16:33:22.078176 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:22.060973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:34605 (local address 127.23.44.62:46671)
0929 16:33:22.061103 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:22.061108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:22.061559 (+ 451us) server_negotiation.cc:407] Connection header received
0929 16:33:22.062797 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:22.062800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:22.062854 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:22.062909 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:22.064419 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:22.065307 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:22.068147 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:22.068543 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:22.068624 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:22.071650 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:22.071655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:22.071658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:22.071685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:22.073958 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:22.074610 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:22.074613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:22.074615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:22.074703 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:22.075102 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:22.075105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:22.075107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:22.075323 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:22.075421 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:22.077883 (+ 2462us) server_negotiation.cc:299] Negotiation successful
0929 16:33:22.078014 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:33:22.079146 3852 heartbeater.cc:347] Connected to a master server at 127.23.44.62:46671
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:33:22.079389 3852 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:22.079952 3852 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
I20260929 16:33:22.081070 3502 ts_manager.cc:194] Registered new tserver with Master: 67f03a8864594abe96c07cf767ffab3b (127.23.44.2:33917)
I20260929 16:33:22.081995 3502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:34605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:33:22.100754 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.44.20:50177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:46671
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:22.262401 3857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:22.262813 3857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:22.262928 3857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:22.268563 3857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:22.268716 3857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:22.268839 3857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:33:22.275615 3857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.44.20:50177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:22.276647 3857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:22.277784 3857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 29 16:33:22 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699601, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
W20260929 16:33:22.304895 3863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:22.305342 3865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:22.305929 3862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:22.306877 3857 server_base.cc:1034] running on GCE node
I20260929 16:33:22.307384 3857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:22.308132 3857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:22.309334 3857 hybrid_clock.cc:648] HybridClock initialized: now 1790699602309271 us; error 73 us; skew 500 ppm
Sep 29 16:33:22 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:22.312687 3857 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:33:22.313696 3857 webserver.cc:460] Webserver started at http://127.23.44.3:36117/ using document root <none> and password file <none>
I20260929 16:33:22.314343 3857 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:22.314411 3857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:22.314615 3857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:33:22 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:33:22.336601 3857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "9849b49eb2c6493487a3e874be8ee446"
format_stamp: "Formatted at 2026-09-29 16:33:22 on dist-test-slave-mlmz"
server_key: "d7e1491817cdcad28a0167ba1465c7cf"
server_key_iv: "90f8ac3517dd860fb775245e62740954"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:22.338341 3857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "9849b49eb2c6493487a3e874be8ee446"
format_stamp: "Formatted at 2026-09-29 16:33:22 on dist-test-slave-mlmz"
server_key: "d7e1491817cdcad28a0167ba1465c7cf"
server_key_iv: "90f8ac3517dd860fb775245e62740954"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:22.345228 3857 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260929 16:33:22.383543 3872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:22.388731 3857 fs_manager.cc:725] Time spent opening block manager: real 0.027s user 0.001s sys 0.000s
I20260929 16:33:22.388911 3857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "9849b49eb2c6493487a3e874be8ee446"
format_stamp: "Formatted at 2026-09-29 16:33:22 on dist-test-slave-mlmz"
server_key: "d7e1491817cdcad28a0167ba1465c7cf"
server_key_iv: "90f8ac3517dd860fb775245e62740954"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:22.389055 3857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:22.417428 3857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:22.421051 3857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:22.421309 3857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:22.422004 3857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:22.423007 3857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:22.423108 3857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:22.423203 3857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:22.423262 3857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:22.433195 3857 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:34963
I20260929 16:33:22.434304 3857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:33:22.435164 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 3857
I20260929 16:33:22.435276 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:33:22.435595 23728 external_mini_cluster.cc:1428] Setting key fdcb63323de7e0f8a02b4d903e4fede5
I20260929 16:33:22.444438 3985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:34963 every 8 connection(s)
Sep 29 16:33:22 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:22.459583 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:22.444178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:58681 (local address 127.23.44.62:46671)
0929 16:33:22.444317 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:22.444322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:22.445039 (+ 717us) server_negotiation.cc:407] Connection header received
0929 16:33:22.446316 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:22.446321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:22.446376 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:22.446431 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:22.448031 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:22.448860 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:22.451831 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:22.452179 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:22.452260 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:22.455365 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:22.455369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:22.455372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:22.455400 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:22.457147 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:22.457790 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:22.457795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:22.457809 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:22.457894 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:22.458295 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:22.458298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:22.458300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:22.458516 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:22.458603 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:22.459295 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 16:33:22.459421 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:33:22.462845 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:22.443097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:46671 (local address 127.23.44.3:58681)
0929 16:33:22.444896 (+ 1799us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:22.444948 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:22.446078 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:22.446836 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:22.446844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:22.447192 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:22.447858 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:22.447879 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:22.449568 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:22.449572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:22.451679 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:22.451690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:22.452461 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:22.452465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:22.452659 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:22.453395 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:22.453417 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:22.455172 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:22.457315 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:22.457322 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:22.457337 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:22.457669 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:22.458003 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:22.458005 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:22.458008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:22.458177 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:22.458674 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:22.458681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:22.459024 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 16:33:22.460278 (+ 1254us) client_negotiation.cc:240] Negotiation successful
0929 16:33:22.461165 (+ 887us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1573,"thread_start_us":165,"threads_started":1}
I20260929 16:33:22.464251 3986 heartbeater.cc:347] Connected to a master server at 127.23.44.62:46671
I20260929 16:33:22.464483 3986 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:22.465014 3986 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
I20260929 16:33:22.466136 3502 ts_manager.cc:194] Registered new tserver with Master: 9849b49eb2c6493487a3e874be8ee446 (127.23.44.3:34963)
I20260929 16:33:22.466830 3502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:58681
I20260929 16:33:22.473420 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:33:22.474218 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 3583
I20260929 16:33:22.482903 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 3722
I20260929 16:33:22.489353 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 3857
I20260929 16:33:22.496191 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 3470
I20260929 16:33:22.503062 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:46671
--webserver_interface=127.23.44.62
--webserver_port=33201
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:46671
--ranger_config_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.44.20:50177/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.44.20:50177/kms with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:22.610895 3989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:22.611155 3989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:22.611231 3989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:22.615206 3989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:22.615288 3989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:22.615332 3989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:22.615389 3989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:22.615429 3989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:22.620554 3989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.44.20:50177/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:46671
--ranger_config_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:46671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=33201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:22.621526 3989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:22.622449 3989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:22.630182 3997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:22.630187 3995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:22.630239 3994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:22.630321 3989 server_base.cc:1034] running on GCE node
I20260929 16:33:22.630977 3989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:22.632014 3989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:22.633175 3989 hybrid_clock.cc:648] HybridClock initialized: now 1790699602633146 us; error 42 us; skew 500 ppm
Sep 29 16:33:22 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:22.635913 3989 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:33:22.636713 3989 webserver.cc:460] Webserver started at http://127.23.44.62:33201/ using document root <none> and password file <none>
I20260929 16:33:22.637220 3989 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:22.637332 3989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:33:22 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:33:22.684361 3989 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:33:22.686751 4005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:22.688129 3989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 16:33:22.688258 3989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "57d735d8c879496c9e7b8e98e9a520f6"
format_stamp: "Formatted at 2026-09-29 16:33:20 on dist-test-slave-mlmz"
server_key: "e8fa8162a0a08f3ab82735f9f9016c3b"
server_key_iv: "5dda0b4dabbe28b975bb69a24b82c39a"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:22.688835 3989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:22.703250 3989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:22.706459 3989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:22.706668 3989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:22.714100 3989 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:46671
I20260929 16:33:22.714131 4057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:46671 every 8 connection(s)
I20260929 16:33:22.715152 3989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:22.719158 4058 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:33:22.719687 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 3989
I20260929 16:33:22.720129 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.44.20:50177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:33803
--local_ip_for_outbound_sockets=127.23.44.1
--tserver_master_addrs=127.23.44.62:46671
--webserver_port=35871
--webserver_interface=127.23.44.1
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.44.20:50177/kms with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:22.720834 4058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6: Bootstrap starting.
I20260929 16:33:22.737051 4058 log.cc:824] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:22.743060 4058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:33:22.743433 4058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6: Bootstrap complete.
I20260929 16:33:22.747381 4058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d735d8c879496c9e7b8e98e9a520f6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46671 } }
I20260929 16:33:22.747789 4058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d735d8c879496c9e7b8e98e9a520f6, State: Initialized, Role: FOLLOWER
I20260929 16:33:22.748236 4058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d735d8c879496c9e7b8e98e9a520f6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46671 } }
I20260929 16:33:22.748342 4058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:22.748435 4058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:22.748545 4058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:33:22.750317 4058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d735d8c879496c9e7b8e98e9a520f6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46671 } }
I20260929 16:33:22.750631 4058 leader_election.cc:302] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57d735d8c879496c9e7b8e98e9a520f6; no voters:
I20260929 16:33:22.750903 4058 leader_election.cc:288] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:33:22.751039 4062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:33:22.751286 4062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [term 2 LEADER]: Becoming Leader. State: Replica: 57d735d8c879496c9e7b8e98e9a520f6, State: Running, Role: LEADER
I20260929 16:33:22.751683 4062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d735d8c879496c9e7b8e98e9a520f6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46671 } }
I20260929 16:33:22.752053 4058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:22.753126 4063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "57d735d8c879496c9e7b8e98e9a520f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d735d8c879496c9e7b8e98e9a520f6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46671 } } }
I20260929 16:33:22.753250 4063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:22.753746 4064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57d735d8c879496c9e7b8e98e9a520f6. Latest consensus state: current_term: 2 leader_uuid: "57d735d8c879496c9e7b8e98e9a520f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d735d8c879496c9e7b8e98e9a520f6" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46671 } } }
I20260929 16:33:22.753894 4064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:22.754274 4071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:22.754271 4058 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:33:22.760141 4071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:22.761070 4071 catalog_manager.cc:1259] Loaded cluster ID: eccdcfef201a458f8f0af3d713b59700
I20260929 16:33:22.761159 4071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:22.764097 4071 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:22.764843 4071 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6: Loaded TSK: 0
I20260929 16:33:22.766187 4071 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:33:22.869215 4060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:22.869580 4060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:22.869647 4060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:22.876183 4060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:22.876298 4060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:22.876434 4060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:33:22.884694 4060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.44.20:50177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:33803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=35871
--tserver_master_addrs=127.23.44.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:22.886152 4060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:22.887481 4060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:22.899583 4096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:22.899575 4095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:22.900110 4098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:22.900722 4060 server_base.cc:1034] running on GCE node
I20260929 16:33:22.901162 4060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:22.901932 4060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:22.903138 4060 hybrid_clock.cc:648] HybridClock initialized: now 1790699602903119 us; error 32 us; skew 500 ppm
Sep 29 16:33:22 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:22.906879 4060 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:33:22.907974 4060 webserver.cc:460] Webserver started at http://127.23.44.1:35871/ using document root <none> and password file <none>
I20260929 16:33:22.908752 4060 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:22.908918 4060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:33:22 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:33:22.936208 4060 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260929 16:33:22.942139 4106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:22.943864 4060 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260929 16:33:22.944005 4060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "5c33f89b507a4ccd9093f8d2be20c9e1"
format_stamp: "Formatted at 2026-09-29 16:33:21 on dist-test-slave-mlmz"
server_key: "eb034099d11507faae7ec782af62bbd3"
server_key_iv: "35da608ab42febfa6f363fceca2e527c"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:22.944744 4060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:22.968650 4060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:22.972630 4060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:22.972895 4060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:22.973632 4060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:22.974694 4060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:22.974802 4060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:22.974925 4060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:22.975005 4060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:22.985808 4060 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:33803
I20260929 16:33:22.987062 4060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:22.990041 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 4060
I20260929 16:33:22.990640 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.44.20:50177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:33917
--local_ip_for_outbound_sockets=127.23.44.2
--tserver_master_addrs=127.23.44.62:46671
--webserver_port=44145
--webserver_interface=127.23.44.2
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.44.20:50177/kms with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:22.992398 4219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:33803 every 8 connection(s)
Sep 29 16:33:23 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699602, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:23.014008 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:22.994956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:44489 (local address 127.23.44.62:46671)
0929 16:33:22.995400 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:22.995422 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:22.998227 (+ 2805us) server_negotiation.cc:407] Connection header received
0929 16:33:22.998578 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:22.998596 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:22.999061 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:22.999464 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:23.000750 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.002291 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.003969 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.004645 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.004808 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:23.007973 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:23.007981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:23.008000 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:23.008032 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:23.010227 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:23.010841 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:23.010846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:23.010852 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:23.011008 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:23.011449 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:23.011454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:23.011457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:23.011935 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:23.012076 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:23.012805 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 16:33:23.013082 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":122,"threads_started":1}
I20260929 16:33:23.014021 4224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:22.995171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:46671 (local address 127.23.44.1:44489)
0929 16:33:22.996386 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:22.996445 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:22.997850 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:22.999548 (+ 1698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:22.999557 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:22.999918 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:23.000547 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:23.000569 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.002443 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.002446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:23.003817 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:23.003828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.005025 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.005029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:23.005215 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:23.005979 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:23.006000 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:23.007743 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:23.010363 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:23.010368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:23.010381 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:23.010706 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:23.011126 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:23.011130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:23.011132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:23.011315 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:23.012078 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:23.012084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:23.012426 (+ 342us) client_negotiation.cc:769] Sending connection context
0929 16:33:23.012680 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 16:33:23.013022 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"thread_start_us":148,"threads_started":1}
I20260929 16:33:23.017047 4220 heartbeater.cc:347] Connected to a master server at 127.23.44.62:46671
I20260929 16:33:23.017323 4220 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:23.017899 4220 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
W20260929 16:33:23.020146 4220 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:46671 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:33:23.020260 4220 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:23.020409 4220 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
I20260929 16:33:23.021337 4220 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:23.021490 4220 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
W20260929 16:33:23.022279 4220 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:33:23.131419 4222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:23.131677 4222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:23.131722 4222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:23.136664 4222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:23.136744 4222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:23.136835 4222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:33:23.145088 4222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.44.20:50177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:33917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=44145
--tserver_master_addrs=127.23.44.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:23.146399 4222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:23.147758 4222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:23.159066 4229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:23.159631 4232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:23.162218 4230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:23.163220 4222 server_base.cc:1034] running on GCE node
I20260929 16:33:23.163656 4222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:23.164376 4222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:23.165555 4222 hybrid_clock.cc:648] HybridClock initialized: now 1790699603165518 us; error 48 us; skew 500 ppm
Sep 29 16:33:23 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699603, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:23.168475 4222 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:33:23.169600 4222 webserver.cc:460] Webserver started at http://127.23.44.2:44145/ using document root <none> and password file <none>
I20260929 16:33:23.170408 4222 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:23.170573 4222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:33:23 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699603, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:33:23.188153 4222 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:33:23.194020 4240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:23.195582 4222 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260929 16:33:23.195719 4222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "67f03a8864594abe96c07cf767ffab3b"
format_stamp: "Formatted at 2026-09-29 16:33:21 on dist-test-slave-mlmz"
server_key: "adc0f7fce6831c0a8555cee2d32c8504"
server_key_iv: "4c9a3b60b1130f110206c808304c78a8"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:23.196307 4222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:23.210688 4222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:23.213708 4222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:23.213903 4222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:23.214498 4222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:23.215510 4222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:23.215615 4222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:23.215714 4222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:23.215781 4222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:23.226593 4222 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:33917
I20260929 16:33:23.227551 4222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:23.228479 4353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:33917 every 8 connection(s)
I20260929 16:33:23.229781 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 4222
I20260929 16:33:23.230260 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.44.20:50177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:34963
--local_ip_for_outbound_sockets=127.23.44.3
--tserver_master_addrs=127.23.44.62:46671
--webserver_port=36117
--webserver_interface=127.23.44.3
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.44.20:50177/kms with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:33:23 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699603, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:23.246043 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:23.232454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:42081 (local address 127.23.44.62:46671)
0929 16:33:23.232632 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:23.232638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:23.232931 (+ 293us) server_negotiation.cc:407] Connection header received
0929 16:33:23.234262 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:23.234266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:23.234331 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:23.234385 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:23.237390 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.238211 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.239627 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.239870 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.239995 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:23.242335 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:23.242339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:23.242342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:23.242369 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:23.243896 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:23.244407 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:23.244410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:23.244412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:23.244493 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:23.244833 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:23.244837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:23.244839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:23.245018 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:23.245120 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:23.245727 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 16:33:23.245867 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:33:23.246868 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:23.231101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:46671 (local address 127.23.44.2:42081)
0929 16:33:23.232778 (+ 1677us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:23.232833 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:23.234000 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:23.234799 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:23.234809 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:23.235139 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:23.235768 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:23.235786 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.238500 (+ 2714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.238503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:23.239496 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:23.239505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.240030 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.240034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:23.240204 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:23.240812 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:23.240831 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:23.242173 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:23.244027 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:23.244032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:23.244047 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:23.244294 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:23.244583 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:23.244587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:23.244589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:23.244725 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:23.245130 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:23.245138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:23.245492 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 16:33:23.245672 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 16:33:23.245966 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1466,"thread_start_us":153,"threads_started":1}
I20260929 16:33:23.248000 4355 heartbeater.cc:347] Connected to a master server at 127.23.44.62:46671
I20260929 16:33:23.248227 4355 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:23.248698 4355 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
W20260929 16:33:23.250164 4355 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:46671 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:33:23.250289 4355 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:23.250443 4355 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
I20260929 16:33:23.251261 4355 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:23.251446 4355 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
W20260929 16:33:23.252151 4355 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:33:23.384706 4358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:23.385097 4358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:23.385212 4358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:23.391281 4358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:23.391355 4358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:23.391436 4358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:33:23.398957 4358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.44.20:50177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:34963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=36117
--tserver_master_addrs=127.23.44.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:23.400418 4358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:23.401723 4358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:23.412689 4364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:23.412727 4366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:23.413102 4363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:23.413431 4358 server_base.cc:1034] running on GCE node
I20260929 16:33:23.413792 4358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:23.414434 4358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:23.415655 4358 hybrid_clock.cc:648] HybridClock initialized: now 1790699603415627 us; error 37 us; skew 500 ppm
Sep 29 16:33:23 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699603, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:23.418619 4358 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:33:23.419615 4358 webserver.cc:460] Webserver started at http://127.23.44.3:36117/ using document root <none> and password file <none>
I20260929 16:33:23.420189 4358 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:23.420264 4358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:33:23 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699603, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:33:23.437062 4358 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:33:23.440196 4374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:23.441754 4358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:33:23.441874 4358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "9849b49eb2c6493487a3e874be8ee446"
format_stamp: "Formatted at 2026-09-29 16:33:22 on dist-test-slave-mlmz"
server_key: "d7e1491817cdcad28a0167ba1465c7cf"
server_key_iv: "90f8ac3517dd860fb775245e62740954"
server_key_version: "kuduclusterkey@0"
I20260929 16:33:23.442561 4358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:23.472506 4358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:23.476109 4358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:23.476352 4358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:23.477092 4358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:23.478196 4358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:23.478261 4358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:23.478315 4358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:23.478391 4358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 16:33:23.489317 4358 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:34963
I20260929 16:33:23.489393 4487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:34963 every 8 connection(s)
I20260929 16:33:23.490456 4358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:33:23.496264 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 4358
Sep 29 16:33:23 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699603, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:23.506110 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:23.492291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:56919 (local address 127.23.44.62:46671)
0929 16:33:23.492451 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:23.492457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:23.493219 (+ 762us) server_negotiation.cc:407] Connection header received
0929 16:33:23.494183 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:23.494187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:23.494242 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:23.494341 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:23.495533 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.496110 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.497784 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.498136 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.498259 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:23.501863 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:23.501869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:23.501871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:23.501905 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:23.503692 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:23.504304 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:23.504308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:23.504310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:23.504405 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:23.504818 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:23.504822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:23.504824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:23.505029 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:23.505123 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:23.505811 (+ 688us) server_negotiation.cc:299] Negotiation successful
0929 16:33:23.505941 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":3840}
I20260929 16:33:23.507539 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:23.492612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:46671 (local address 127.23.44.3:56919)
0929 16:33:23.493054 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:23.493103 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:23.493984 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:23.494511 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:23.494517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:23.494849 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:23.495387 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:23.495400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.496256 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.496260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:23.497647 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:23.497660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.498307 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.498312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:23.498505 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:23.499212 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:23.499233 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:23.501667 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:23.503820 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:23.503826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:23.503841 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:23.504183 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:23.504509 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:23.504512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:23.504514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:23.504699 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:23.505150 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:23.505156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:23.505515 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 16:33:23.506264 (+ 749us) client_negotiation.cc:240] Negotiation successful
0929 16:33:23.506575 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":147,"threads_started":1}
I20260929 16:33:23.508808 4488 heartbeater.cc:347] Connected to a master server at 127.23.44.62:46671
I20260929 16:33:23.509057 4488 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:23.509629 4488 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
W20260929 16:33:23.511482 4488 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:46671 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:33:23.511637 4488 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:23.511801 4488 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
I20260929 16:33:23.512647 4488 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:23.512809 4488 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
I20260929 16:33:23.513731 4022 ts_manager.cc:194] Registered new tserver with Master: 9849b49eb2c6493487a3e874be8ee446 (127.23.44.3:34963)
I20260929 16:33:23.515249 4022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:56919
I20260929 16:33:23.519156 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:23.496878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.23.44.62:46671)
0929 16:33:23.498364 (+ 1486us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:23.498368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:23.498379 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:33:23.498415 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:23.498417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:23.498447 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:23.498543 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:23.499415 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.499983 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.501323 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:23.501679 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:23.501799 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:23.507727 (+ 5928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:23.507734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:23.507739 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:23.507765 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:23.511040 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:23.517937 (+ 6897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:23.517941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:23.517943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:23.518035 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:23.518409 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:23.518411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:23.518413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:23.518571 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:23.518699 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:23.518918 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 16:33:23.519016 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1418,"thread_start_us":138,"threads_started":1}
Sep 29 16:33:23 dist-test-slave-mlmz krb5kdc[3026](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699603, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:24.022485 4220 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:24.022778 4220 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
I20260929 16:33:24.023453 4022 ts_manager.cc:194] Registered new tserver with Master: 5c33f89b507a4ccd9093f8d2be20c9e1 (127.23.44.1:33803)
I20260929 16:33:24.024031 4022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:44489
Sep 29 16:33:24 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699603, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:33:24.252410 4355 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:24.252672 4355 heartbeater.cc:511] Master 127.23.44.62:46671 requested a full tablet report, sending...
I20260929 16:33:24.253512 4022 ts_manager.cc:194] Registered new tserver with Master: 67f03a8864594abe96c07cf767ffab3b (127.23.44.2:33917)
I20260929 16:33:24.254264 4022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:42081
I20260929 16:33:24.255209 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:33:24.266391 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.256982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.23.44.62:46671)
0929 16:33:24.257411 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:24.257415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:24.257430 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:33:24.257605 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:24.257607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:24.257651 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:24.257723 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:24.258575 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.259100 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.259899 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.260128 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.260236 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:24.261597 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:24.261600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:24.261603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:24.261628 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:24.263446 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:24.263943 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:24.263947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:24.263949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:24.264033 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:24.264313 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:24.264317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:24.264319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:24.264505 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:24.264627 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:24.266096 (+ 1469us) server_negotiation.cc:299] Negotiation successful
0929 16:33:24.266217 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":56,"threads_started":1}
I20260929 16:33:24.270043 4021 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:33:24.274406 4021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:33:24.303254 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.293109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:33917 (local address 127.0.0.1:38310)
0929 16:33:24.294300 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:24.294348 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:24.294766 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:24.297509 (+ 2743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:24.297515 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:24.297538 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:24.297679 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:24.297686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.299937 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.299941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:24.301558 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:24.301569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.302698 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.302702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:24.302826 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:24.302878 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 16:33:24.302968 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:33:24.303093 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"thread_start_us":87,"threads_started":1}
I20260929 16:33:24.304299 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.293889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.23.44.2:33917)
0929 16:33:24.297014 (+ 3125us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:24.297021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:24.297043 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:33:24.297114 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:24.297119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:24.297348 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:24.298106 (+ 758us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:24.298263 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.299810 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.301752 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.302550 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.303861 (+ 1311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:24.304011 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:24.304064 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 16:33:24.304166 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3023,"thread_start_us":67,"threads_started":1}
I20260929 16:33:24.308727 4279 tablet_service.cc:1467] Processing CreateTablet for tablet 40f1248490a4447d9ce63219f6a6c2f3 (DEFAULT_TABLE table=test-table [id=656a5d3636424667a3589117b659070d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:24.309955 4279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f1248490a4447d9ce63219f6a6c2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:24.310324 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.295209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:33803 (local address 127.0.0.1:33178)
0929 16:33:24.303408 (+ 8199us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:24.303421 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:24.303494 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:24.305424 (+ 1930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:24.305427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:24.305444 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:24.305516 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:24.305522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.307535 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.307538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:24.308621 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:24.308630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.309922 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.309926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:24.310017 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:24.310045 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:33:24.310099 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 16:33:24.310193 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8145,"thread_start_us":79,"threads_started":1}
I20260929 16:33:24.311004 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.295977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.23.44.1:33803)
0929 16:33:24.304674 (+ 8697us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:24.304682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:24.304706 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:33:24.304778 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:24.304784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:24.305011 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:24.306148 (+ 1137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:24.306290 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.307398 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.308789 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.309772 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.310578 (+ 806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:24.310721 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:24.310772 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 16:33:24.310870 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8589,"thread_start_us":101,"threads_started":1}
I20260929 16:33:24.315950 4152 tablet_service.cc:1467] Processing CreateTablet for tablet 40f1248490a4447d9ce63219f6a6c2f3 (DEFAULT_TABLE table=test-table [id=656a5d3636424667a3589117b659070d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:24.316977 4152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f1248490a4447d9ce63219f6a6c2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:24.324939 4521 tablet_bootstrap.cc:492] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b: Bootstrap starting.
I20260929 16:33:24.327733 4521 tablet_bootstrap.cc:654] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:24.328613 4521 log.cc:824] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:24.331161 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.294893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:34963 (local address 127.0.0.1:52930)
0929 16:33:24.295204 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:24.295217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:24.295297 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:24.326348 (+ 31051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:24.326353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:24.326372 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:24.326478 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:24.326484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.328087 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.328090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:24.329577 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:24.329588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.330719 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.330722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:24.330838 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:24.330870 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:33:24.330926 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 16:33:24.331009 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":86,"threads_started":1}
I20260929 16:33:24.332139 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.295267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.23.44.3:34963)
0929 16:33:24.325876 (+ 30609us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:24.325883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:24.325909 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:33:24.325985 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:24.325990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:24.326185 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:24.326637 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:24.326783 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.327941 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.329738 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.330582 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.331627 (+ 1045us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:24.331769 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:24.331830 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 16:33:24.331950 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30478,"thread_start_us":119,"threads_started":1}
I20260929 16:33:24.335098 4414 tablet_service.cc:1467] Processing CreateTablet for tablet 40f1248490a4447d9ce63219f6a6c2f3 (DEFAULT_TABLE table=test-table [id=656a5d3636424667a3589117b659070d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:24.336083 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f1248490a4447d9ce63219f6a6c2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:24.337579 4522 tablet_bootstrap.cc:492] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1: Bootstrap starting.
I20260929 16:33:24.339710 4522 tablet_bootstrap.cc:654] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:24.340529 4522 log.cc:824] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:24.349913 4521 tablet_bootstrap.cc:492] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b: No bootstrap required, opened a new log
I20260929 16:33:24.350176 4521 ts_tablet_manager.cc:1397] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20260929 16:33:24.352555 4522 tablet_bootstrap.cc:492] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1: No bootstrap required, opened a new log
I20260929 16:33:24.352823 4522 ts_tablet_manager.cc:1397] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260929 16:33:24.354552 4521 raft_consensus.cc:348] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f03a8864594abe96c07cf767ffab3b" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 33917 } } peers { permanent_uuid: "9849b49eb2c6493487a3e874be8ee446" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34963 } } peers { permanent_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 33803 } }
I20260929 16:33:24.354867 4524 tablet_bootstrap.cc:492] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446: Bootstrap starting.
I20260929 16:33:24.354911 4521 raft_consensus.cc:374] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:24.355041 4521 raft_consensus.cc:729] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f03a8864594abe96c07cf767ffab3b, State: Initialized, Role: FOLLOWER
I20260929 16:33:24.355573 4521 consensus_queue.cc:260] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f03a8864594abe96c07cf767ffab3b" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 33917 } } peers { permanent_uuid: "9849b49eb2c6493487a3e874be8ee446" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34963 } } peers { permanent_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 33803 } }
I20260929 16:33:24.356837 4521 ts_tablet_manager.cc:1428] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:33:24.356901 4522 raft_consensus.cc:348] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f03a8864594abe96c07cf767ffab3b" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 33917 } } peers { permanent_uuid: "9849b49eb2c6493487a3e874be8ee446" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34963 } } peers { permanent_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 33803 } }
I20260929 16:33:24.357201 4355 heartbeater.cc:503] Master 127.23.44.62:46671 was elected leader, sending a full tablet report...
I20260929 16:33:24.357197 4522 raft_consensus.cc:374] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:24.357287 4522 raft_consensus.cc:729] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c33f89b507a4ccd9093f8d2be20c9e1, State: Initialized, Role: FOLLOWER
I20260929 16:33:24.357280 4524 tablet_bootstrap.cc:654] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:24.357746 4522 consensus_queue.cc:260] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f03a8864594abe96c07cf767ffab3b" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 33917 } } peers { permanent_uuid: "9849b49eb2c6493487a3e874be8ee446" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34963 } } peers { permanent_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 33803 } }
I20260929 16:33:24.358140 4524 log.cc:824] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:24.358739 4522 ts_tablet_manager.cc:1428] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:33:24.365414 4220 heartbeater.cc:503] Master 127.23.44.62:46671 was elected leader, sending a full tablet report...
I20260929 16:33:24.368528 4524 tablet_bootstrap.cc:492] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446: No bootstrap required, opened a new log
I20260929 16:33:24.368711 4524 ts_tablet_manager.cc:1397] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260929 16:33:24.372792 4524 raft_consensus.cc:348] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f03a8864594abe96c07cf767ffab3b" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 33917 } } peers { permanent_uuid: "9849b49eb2c6493487a3e874be8ee446" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34963 } } peers { permanent_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 33803 } }
I20260929 16:33:24.373128 4524 raft_consensus.cc:374] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:24.373222 4524 raft_consensus.cc:729] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9849b49eb2c6493487a3e874be8ee446, State: Initialized, Role: FOLLOWER
I20260929 16:33:24.373788 4524 consensus_queue.cc:260] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f03a8864594abe96c07cf767ffab3b" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 33917 } } peers { permanent_uuid: "9849b49eb2c6493487a3e874be8ee446" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34963 } } peers { permanent_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 33803 } }
I20260929 16:33:24.374847 4524 ts_tablet_manager.cc:1428] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:33:24.375226 4488 heartbeater.cc:503] Master 127.23.44.62:46671 was elected leader, sending a full tablet report...
W20260929 16:33:24.480017 4356 tablet.cc:2369] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:33:24.491997 4489 tablet.cc:2369] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:33:24.494850 4221 tablet.cc:2369] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 16:33:24 dist-test-slave-mlmz krb5kdc[3026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699603, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for HTTP/127.23.44.20@KRBTEST.COM
I20260929 16:33:24.596486 4527 raft_consensus.cc:482] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:24.596729 4527 raft_consensus.cc:504] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f03a8864594abe96c07cf767ffab3b" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 33917 } } peers { permanent_uuid: "9849b49eb2c6493487a3e874be8ee446" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34963 } } peers { permanent_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 33803 } }
I20260929 16:33:24.598059 4527 leader_election.cc:288] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67f03a8864594abe96c07cf767ffab3b (127.23.44.2:33917), 9849b49eb2c6493487a3e874be8ee446 (127.23.44.3:34963)
I20260929 16:33:24.603818 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.598653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:46953 (local address 127.23.44.2:33917)
0929 16:33:24.598824 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:24.598829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:24.599198 (+ 369us) server_negotiation.cc:407] Connection header received
0929 16:33:24.599422 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:24.599426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:24.599480 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:24.599530 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:24.599979 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.601007 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.602605 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.603181 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.603257 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:24.603318 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:24.603627 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 16:33:24.603694 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:33:24.604491 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.598204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:33917 (local address 127.23.44.1:46953)
0929 16:33:24.599105 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:24.599120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:24.599289 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:24.599678 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:24.599681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:24.599698 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:24.599853 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:24.599861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.601139 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.601142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:24.602463 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:24.602474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.603409 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.603412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:24.603504 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:24.603525 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:33:24.604204 (+ 679us) client_negotiation.cc:240] Negotiation successful
0929 16:33:24.604338 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":47,"thread_start_us":129,"threads_started":1}
I20260929 16:33:24.605232 4296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f1248490a4447d9ce63219f6a6c2f3" candidate_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f03a8864594abe96c07cf767ffab3b" is_pre_election: true
I20260929 16:33:24.605598 4296 raft_consensus.cc:2454] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c33f89b507a4ccd9093f8d2be20c9e1 in term 0.
I20260929 16:33:24.606096 4109 leader_election.cc:302] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c33f89b507a4ccd9093f8d2be20c9e1, 67f03a8864594abe96c07cf767ffab3b; no voters:
I20260929 16:33:24.606383 4527 raft_consensus.cc:2781] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:33:24.606455 4527 raft_consensus.cc:482] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:33:24.606494 4527 raft_consensus.cc:3037] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:24.606820 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.598811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:51039 (local address 127.23.44.3:34963)
0929 16:33:24.598949 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:24.598953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:24.601109 (+ 2156us) server_negotiation.cc:407] Connection header received
0929 16:33:24.601804 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:24.601808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:24.601882 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:24.601932 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:24.603194 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.603935 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.605509 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.606136 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.606218 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:24.606277 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:24.606628 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 16:33:24.606707 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:33:24.607215 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.598320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:34963 (local address 127.23.44.1:51039)
0929 16:33:24.601022 (+ 2702us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:24.601037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:24.601394 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:24.602922 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:24.602925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:24.602943 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:24.603057 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:24.603064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.604083 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.604086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:24.605367 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:24.605378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.606384 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.606387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:24.606489 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:24.606510 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:33:24.606995 (+ 485us) client_negotiation.cc:240] Negotiation successful
0929 16:33:24.607067 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2252,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260929 16:33:24.607730 4426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f1248490a4447d9ce63219f6a6c2f3" candidate_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9849b49eb2c6493487a3e874be8ee446" is_pre_election: true
I20260929 16:33:24.607779 4527 raft_consensus.cc:504] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f03a8864594abe96c07cf767ffab3b" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 33917 } } peers { permanent_uuid: "9849b49eb2c6493487a3e874be8ee446" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34963 } } peers { permanent_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 33803 } }
I20260929 16:33:24.607976 4426 raft_consensus.cc:2454] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c33f89b507a4ccd9093f8d2be20c9e1 in term 0.
I20260929 16:33:24.608124 4527 leader_election.cc:288] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [CANDIDATE]: Term 1 election: Requested vote from peers 67f03a8864594abe96c07cf767ffab3b (127.23.44.2:33917), 9849b49eb2c6493487a3e874be8ee446 (127.23.44.3:34963)
I20260929 16:33:24.608803 4426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f1248490a4447d9ce63219f6a6c2f3" candidate_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9849b49eb2c6493487a3e874be8ee446"
I20260929 16:33:24.608915 4426 raft_consensus.cc:3037] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:24.608913 4296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f1248490a4447d9ce63219f6a6c2f3" candidate_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f03a8864594abe96c07cf767ffab3b"
I20260929 16:33:24.609022 4296 raft_consensus.cc:3037] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:24.610092 4426 raft_consensus.cc:2454] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c33f89b507a4ccd9093f8d2be20c9e1 in term 1.
I20260929 16:33:24.610277 4296 raft_consensus.cc:2454] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c33f89b507a4ccd9093f8d2be20c9e1 in term 1.
I20260929 16:33:24.610565 4109 leader_election.cc:302] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c33f89b507a4ccd9093f8d2be20c9e1, 67f03a8864594abe96c07cf767ffab3b; no voters:
I20260929 16:33:24.610744 4527 raft_consensus.cc:2781] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:24.610944 4527 raft_consensus.cc:686] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [term 1 LEADER]: Becoming Leader. State: Replica: 5c33f89b507a4ccd9093f8d2be20c9e1, State: Running, Role: LEADER
I20260929 16:33:24.611258 4527 consensus_queue.cc:237] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f03a8864594abe96c07cf767ffab3b" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 33917 } } peers { permanent_uuid: "9849b49eb2c6493487a3e874be8ee446" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34963 } } peers { permanent_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 33803 } }
I20260929 16:33:24.615880 4021 catalog_manager.cc:5495] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c33f89b507a4ccd9093f8d2be20c9e1 (127.23.44.1). New cstate: current_term: 1 leader_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f03a8864594abe96c07cf767ffab3b" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9849b49eb2c6493487a3e874be8ee446" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c33f89b507a4ccd9093f8d2be20c9e1" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 33803 } health_report { overall_health: HEALTHY } } }
I20260929 16:33:24.680066 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.675197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.23.44.1:33803)
0929 16:33:24.675343 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:24.675348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:24.675362 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:33:24.675463 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:24.675467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:24.675534 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:24.675635 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:33:24.675881 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.676697 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.677968 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:24.678308 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:24.678422 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:24.678816 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:33:24.679344 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:33:24.679455 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:24.679843 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 16:33:24.679904 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:33:24.703075 4426 raft_consensus.cc:1261] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446 [term 1 FOLLOWER]: Refusing update from remote peer 5c33f89b507a4ccd9093f8d2be20c9e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:24.703958 4533 consensus_queue.cc:1035] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9849b49eb2c6493487a3e874be8ee446" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 34963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:24.709761 4296 raft_consensus.cc:1261] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b [term 1 FOLLOWER]: Refusing update from remote peer 5c33f89b507a4ccd9093f8d2be20c9e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:24.710479 4533 consensus_queue.cc:1035] T 40f1248490a4447d9ce63219f6a6c2f3 P 5c33f89b507a4ccd9093f8d2be20c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67f03a8864594abe96c07cf767ffab3b" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:24.728797 4537 mvcc.cc:205] Tried to move back new op lower bound from 7334705580848091136 to 7334705580491190272. Current Snapshot: MvccSnapshot[applied={T|T < 7334705580848091136}]
I20260929 16:33:24.737170 4540 mvcc.cc:205] Tried to move back new op lower bound from 7334705580848091136 to 7334705580491190272. Current Snapshot: MvccSnapshot[applied={T|T < 7334705580848091136 or (T in {7334705580848091136})}]
I20260929 16:33:24.739635 4538 mvcc.cc:205] Tried to move back new op lower bound from 7334705580848091136 to 7334705580491190272. Current Snapshot: MvccSnapshot[applied={T|T < 7334705580848091136}]
I20260929 16:33:24.747259 4021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:24.747417 4021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:24.750169 4021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57d735d8c879496c9e7b8e98e9a520f6: Sending DeleteTablet for 3 replicas of tablet 40f1248490a4447d9ce63219f6a6c2f3
I20260929 16:33:24.751364 4279 tablet_service.cc:1514] Processing DeleteTablet for tablet 40f1248490a4447d9ce63219f6a6c2f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:24 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:38310
I20260929 16:33:24.751524 4414 tablet_service.cc:1514] Processing DeleteTablet for tablet 40f1248490a4447d9ce63219f6a6c2f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:24 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:52930
I20260929 16:33:24.751642 4152 tablet_service.cc:1514] Processing DeleteTablet for tablet 40f1248490a4447d9ce63219f6a6c2f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:24 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:33178
I20260929 16:33:24.752938 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 4060
I20260929 16:33:24.761499 4546 tablet_replica.cc:330] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b: stopping tablet replica
I20260929 16:33:24.761891 4546 raft_consensus.cc:2229] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:24.762290 4546 raft_consensus.cc:2258] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:24.763486 4547 tablet_replica.cc:330] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446: stopping tablet replica
I20260929 16:33:24.763818 4547 raft_consensus.cc:2229] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:24.764855 4547 raft_consensus.cc:2258] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:24.766106 4546 ts_tablet_manager.cc:1905] T 40f1248490a4447d9ce63219f6a6c2f3 P 67f03a8864594abe96c07cf767ffab3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:24.766466 4547 ts_tablet_manager.cc:1905] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:24.769137 4547 ts_tablet_manager.cc:1918] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 16:33:24.769580 4007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:33803 (error 108)
I20260929 16:33:24.770275 4547 log.cc:1196] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/40f1248490a4447d9ce63219f6a6c2f3
I20260929 16:33:24.770517 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 4222
I20260929 16:33:24.779253 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 4358
I20260929 16:33:24.779364 4547 ts_tablet_manager.cc:1939] T 40f1248490a4447d9ce63219f6a6c2f3 P 9849b49eb2c6493487a3e874be8ee446: Deleting consensus metadata
I20260929 16:33:24.780980 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.780641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:33803 (local address 127.0.0.1:33196)
0929 16:33:24.780813 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:24.780891 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 16:33:24.781149 4007 catalog_manager.cc:4584] TS 5c33f89b507a4ccd9093f8d2be20c9e1 (127.23.44.1:33803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40f1248490a4447d9ce63219f6a6c2f3: Network error: Client connection negotiation failed: client connection to 127.23.44.1:33803: connect: Connection refused (error 111)
I20260929 16:33:24.782097 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.781828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:33917 (local address 127.0.0.1:38322)
0929 16:33:24.781986 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:24.782033 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.2:33917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 16:33:24.787875 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.787652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:34963 (local address 127.0.0.1:52936)
0929 16:33:24.787757 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:24.787812 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.3:34963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 16:33:24.788019 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 3989
I20260929 16:33:24.796218 23728 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:33:25.140066 23728 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:33:25.140178 23728 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:33:25.484341 23728 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:33:25.484 UTC [3056] LOG: received smart shutdown request
2026-09-29 16:33:25.488 UTC [3056] LOG: background worker "logical replication launcher" (PID 3065) exited with exit code 1
2026-09-29 16:33:25.488 UTC [3060] LOG: shutting down
2026-09-29 16:33:25.666 UTC [3056] LOG: database system is shut down
2026-09-29T16:33:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48071 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:33:25 dist-test-slave-mlmz krb5kdc[4558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:33:25 dist-test-slave-mlmz krb5kdc[4558](info): set up 2 sockets
Sep 29 16:33:25 dist-test-slave-mlmz krb5kdc[4558](info): commencing operation
krb5kdc: starting...
W20260929 16:33:26.829784 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:33:26 dist-test-slave-mlmz krb5kdc[4558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:33:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:33:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:33:26.948004 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39765
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:39765,127.23.44.61:33069,127.23.44.60:42687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:27.055224 4571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:27.055444 4571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:27.055481 4571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:27.059396 4571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:27.059441 4571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:27.059460 4571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:27.059476 4571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:27.059515 4571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:27.064527 4571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:39765,127.23.44.61:33069,127.23.44.60:42687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:27.065531 4571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:27.066437 4571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:27.074144 4579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:27.074105 4577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:27.074184 4571 server_base.cc:1034] running on GCE node
W20260929 16:33:27.074141 4576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:27.074846 4571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:27.075865 4571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:27.077066 4571 hybrid_clock.cc:648] HybridClock initialized: now 1790699607077037 us; error 29 us; skew 500 ppm
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:27.079757 4571 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:33:27.080489 4571 webserver.cc:460] Webserver started at http://127.23.44.62:41997/ using document root <none> and password file <none>
I20260929 16:33:27.080950 4571 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:27.081004 4571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:27.081151 4571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:27.082996 4571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "612c059780bd409eaa96898c59eac9b2"
format_stamp: "Formatted at 2026-09-29 16:33:27 on dist-test-slave-mlmz"
server_key: "59ca0ff0279f1ab658b8887f329f80f1"
server_key_iv: "16228c48bfafbdd2aeee7c95f50c9eb5"
server_key_version: "encryptionkey@0"
I20260929 16:33:27.083469 4571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "612c059780bd409eaa96898c59eac9b2"
format_stamp: "Formatted at 2026-09-29 16:33:27 on dist-test-slave-mlmz"
server_key: "59ca0ff0279f1ab658b8887f329f80f1"
server_key_iv: "16228c48bfafbdd2aeee7c95f50c9eb5"
server_key_version: "encryptionkey@0"
I20260929 16:33:27.086817 4571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:33:27.088943 4586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:27.090227 4571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:33:27.090365 4571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "612c059780bd409eaa96898c59eac9b2"
format_stamp: "Formatted at 2026-09-29 16:33:27 on dist-test-slave-mlmz"
server_key: "59ca0ff0279f1ab658b8887f329f80f1"
server_key_iv: "16228c48bfafbdd2aeee7c95f50c9eb5"
server_key_version: "encryptionkey@0"
I20260929 16:33:27.090472 4571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:27.104521 4571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:27.107442 4571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:27.107649 4571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:27.114639 4571 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:39765
I20260929 16:33:27.114646 4638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:39765 every 8 connection(s)
I20260929 16:33:27.115559 4571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:27.118093 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:27.121508 4639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:33:27.123116 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 4571
I20260929 16:33:27.123251 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:33:27.123535 23728 external_mini_cluster.cc:1428] Setting key 73e025da0db5309c7292a25518b5aadb
WARNING: no policy specified for kudu/127.23.44.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.61@KRBTEST.COM" created.
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:27.139727 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.124343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.23.44.62:39765)
0929 16:33:27.124584 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.124593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.124625 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:33:27.125702 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.125708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.125982 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.126189 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.126891 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.128322 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.129293 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.129915 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.130086 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.134793 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.134798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.134807 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.134826 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.136808 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.137291 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.137297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.137298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.137399 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.137708 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.137711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.137713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.138048 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.138183 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.138525 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.138756 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260929 16:33:27.139727 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.123674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39765 (local address 127.0.0.1:54378)
0929 16:33:27.124161 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.124217 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:27.125402 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:27.126197 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:27.126202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:27.126515 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:27.126737 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.126748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.128445 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.128447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.129141 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.129150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.130093 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.130096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.130267 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.130900 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:27.130916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:27.134633 (+ 3717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:27.136911 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.136921 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.136930 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.137157 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.137504 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.137507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.137509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.137612 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.138167 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:27.138173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:27.138304 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 16:33:27.138466 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 16:33:27.138815 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":156,"thread_start_us":94,"threads_started":1}
I20260929 16:33:27.142241 4639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:33:27.142985 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.142679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:33069 (local address 127.0.0.1:46996)
0929 16:33:27.142855 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.142930 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 16:33:27.143196 4588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.44.61:33069: connect: Connection refused (error 111)
I20260929 16:33:27.144062 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.143748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:33069 (local address 127.0.0.1:47002)
0929 16:33:27.143873 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.143981 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 16:33:27.144258 4639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.61:33069: Network error: Client connection negotiation failed: client connection to 127.23.44.61:33069: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.61.keytab.
Entry for principal kudu/127.23.44.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.61.keytab.
I20260929 16:33:27.149114 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.61:33069
--webserver_interface=127.23.44.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.61.keytab
--principal=kudu/127.23.44.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:39765,127.23.44.61:33069,127.23.44.60:42687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:27.193521 4639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } attempt: 1
I20260929 16:33:27.194439 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.194071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:33069 (local address 127.0.0.1:47012)
0929 16:33:27.194283 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.194351 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 16:33:27.195160 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.194916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:33069 (local address 127.0.0.1:47028)
0929 16:33:27.195022 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.195077 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 16:33:27.195304 4639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.61:33069: Network error: Client connection negotiation failed: client connection to 127.23.44.61:33069: connect: Connection refused (error 111)
W20260929 16:33:27.252461 4646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:27.252677 4646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:27.252714 4646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:27.256605 4646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:27.256655 4646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:27.256675 4646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:27.256690 4646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:27.256704 4646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:27.261536 4646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:39765,127.23.44.61:33069,127.23.44.60:42687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.61.keytab
--principal=kudu/127.23.44.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.61:33069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.44.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:27.262311 4646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:27.263144 4646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:33:27.263474 4639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } attempt: 2
I20260929 16:33:27.264240 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.263937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:33069 (local address 127.0.0.1:47042)
0929 16:33:27.264092 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.264157 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 16:33:27.264961 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.264720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:33069 (local address 127.0.0.1:47056)
0929 16:33:27.264836 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.264885 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:33069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:33:27.265106 4639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.61:33069: Network error: Client connection negotiation failed: client connection to 127.23.44.61:33069: connect: Connection refused (error 111)
W20260929 16:33:27.271301 4652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:27.271327 4651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:27.271312 4654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:27.271572 4646 server_base.cc:1034] running on GCE node
I20260929 16:33:27.272104 4646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:27.273084 4646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:27.274294 4646 hybrid_clock.cc:648] HybridClock initialized: now 1790699607274290 us; error 53 us; skew 500 ppm
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:27.276996 4646 init.cc:377] Logged in from keytab as kudu/127.23.44.61@KRBTEST.COM (short username kudu)
I20260929 16:33:27.277846 4646 webserver.cc:460] Webserver started at http://127.23.44.61:32855/ using document root <none> and password file <none>
I20260929 16:33:27.278345 4646 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:27.278429 4646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:27.278611 4646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:27.280422 4646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/data/instance:
uuid: "79504bb242d64d698415415b882e6282"
format_stamp: "Formatted at 2026-09-29 16:33:27 on dist-test-slave-mlmz"
server_key: "d8e2cd37409a47ece79aff19b2f96f5a"
server_key_iv: "867b966c4a7661328407447bad19269d"
server_key_version: "encryptionkey@0"
I20260929 16:33:27.280934 4646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/wal/instance:
uuid: "79504bb242d64d698415415b882e6282"
format_stamp: "Formatted at 2026-09-29 16:33:27 on dist-test-slave-mlmz"
server_key: "d8e2cd37409a47ece79aff19b2f96f5a"
server_key_iv: "867b966c4a7661328407447bad19269d"
server_key_version: "encryptionkey@0"
I20260929 16:33:27.284301 4646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:33:27.286486 4661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:27.287612 4646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:33:27.287761 4646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/wal
uuid: "79504bb242d64d698415415b882e6282"
format_stamp: "Formatted at 2026-09-29 16:33:27 on dist-test-slave-mlmz"
server_key: "d8e2cd37409a47ece79aff19b2f96f5a"
server_key_iv: "867b966c4a7661328407447bad19269d"
server_key_version: "encryptionkey@0"
I20260929 16:33:27.287864 4646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:27.308480 4646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:27.311474 4646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:27.311661 4646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:27.318105 4646 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.61:33069
I20260929 16:33:27.318166 4713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.61:33069 every 8 connection(s)
I20260929 16:33:27.319013 4646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
I20260929 16:33:27.321576 4714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:27.324287 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 4646
I20260929 16:33:27.324395 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-1/wal/instance
I20260929 16:33:27.324682 23728 external_mini_cluster.cc:1428] Setting key f2c8e71d6ab06dc6cdb0d53398d34570
I20260929 16:33:27.325124 4714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.44.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.60@KRBTEST.COM" created.
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.61@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:27.340754 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.327136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.23.44.62:39765)
0929 16:33:27.327295 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.327302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.328070 (+ 768us) server_negotiation.cc:407] Connection header received
0929 16:33:27.329362 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.329365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.329410 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.329481 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.331048 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.331567 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.333292 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.333547 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.333638 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.336754 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.336757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.336759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.336782 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.338493 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.338980 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.338987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.338992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.339085 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.339513 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.339519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.339523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.339784 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.339907 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.340377 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.340517 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:33:27.341308 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.327432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39765 (local address 127.0.0.1:54394)
0929 16:33:27.327897 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.327951 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:27.329101 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:27.329695 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:27.329706 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:27.330209 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:27.330890 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.330912 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.331698 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.331702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.333133 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.333145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.333680 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.333683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.333815 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.334394 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:27.334410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:27.336574 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:27.338612 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.338616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.338626 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.338878 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.339194 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.339197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.339198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.339372 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.339908 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:27.339912 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:27.340124 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 16:33:27.340304 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 16:33:27.340521 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":96,"threads_started":1}
I20260929 16:33:27.342638 4714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.61@KRBTEST.COM for kudu/127.23.44.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.60.keytab.
Entry for principal kudu/127.23.44.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.60.keytab.
I20260929 16:33:27.351475 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.343088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:33069 (local address 127.0.0.1:47072)
0929 16:33:27.343192 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.343206 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:27.343273 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:27.344078 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:27.344080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:27.344203 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:27.344283 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.344287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.345348 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.345354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.345869 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.345877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.346500 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.346504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.346588 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.347057 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:27.347075 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:27.348195 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:27.350010 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.350013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.350014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.350193 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.350513 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.350515 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.350517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.350590 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.351019 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:27.351021 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:27.351098 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 16:33:27.351177 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:33:27.351327 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 16:33:27.351706 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.343239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47072 (local address 127.23.44.61:33069)
0929 16:33:27.343622 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.343633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.343655 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:33:27.343703 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.343709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.343915 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.344047 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.344393 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.345152 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.345988 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.346325 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.346479 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.348315 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.348323 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.348332 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.348352 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.349904 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.350286 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.350290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.350291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.350423 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.350670 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.350672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.350673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.350934 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.351031 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.351245 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.351410 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":196,"threads_started":1}
I20260929 16:33:27.352358 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.60:42687
--webserver_interface=127.23.44.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.60.keytab
--principal=kudu/127.23.44.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:39765,127.23.44.61:33069,127.23.44.60:42687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:27.353318 4714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:33:27.354014 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.353724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60762)
0929 16:33:27.353855 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.353942 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:33:27.354223 4663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
I20260929 16:33:27.354981 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.354795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60764)
0929 16:33:27.354896 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.354937 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 16:33:27.355146 4714 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:42687: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
I20260929 16:33:27.356276 4639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } attempt: 3
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.61@KRBTEST.COM
I20260929 16:33:27.364738 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.356748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.23.44.61:33069)
0929 16:33:27.356852 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.356855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.356867 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:33:27.357012 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.357014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.357047 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.357107 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.357661 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.358138 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.359355 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.359570 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.359643 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.361860 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.361863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.361865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.361884 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.363355 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.363790 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.363792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.363793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.363858 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.364110 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.364112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.364113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.364251 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.364320 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.364573 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.364651 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 16:33:27.364804 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.356644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:33069 (local address 127.0.0.1:47082)
0929 16:33:27.356799 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.356816 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:27.356919 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:27.357150 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:27.357155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:27.357393 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:27.357546 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.357557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.358246 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.358251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.359235 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.359251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.359682 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.359688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.359825 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.360262 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:27.360282 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:27.361705 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:27.363460 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.363463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.363466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.363698 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.363939 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.363942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.363944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.364028 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.364334 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:27.364337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:27.364427 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 16:33:27.364529 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:33:27.364646 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:33:27.365883 4639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:33:27.366521 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.366307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60780)
0929 16:33:27.366426 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.366469 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:33:27.367228 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.366980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60794)
0929 16:33:27.367132 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.367177 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260929 16:33:27.367436 4639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:42687: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
I20260929 16:33:27.398595 4639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } attempt: 1
I20260929 16:33:27.399255 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.399023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60810)
0929 16:33:27.399142 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.399193 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:33:27.399899 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.399694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60826)
0929 16:33:27.399799 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.399851 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:33:27.400022 4639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:42687: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
I20260929 16:33:27.404371 4714 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } attempt: 1
I20260929 16:33:27.405007 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.404785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60840)
0929 16:33:27.404901 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.404949 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:33:27.405675 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.405467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60848)
0929 16:33:27.405574 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.405624 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:33:27.405797 4714 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:42687: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
W20260929 16:33:27.461000 4720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:27.461257 4720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:27.461359 4720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:27.465205 4720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:27.465306 4720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:27.465379 4720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:27.465417 4720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:27.465448 4720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:27.470459 4720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:39765,127.23.44.61:33069,127.23.44.60:42687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.60.keytab
--principal=kudu/127.23.44.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.60:42687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.44.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:27.471390 4720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:27.472293 4720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:33:27.474043 4714 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } attempt: 2
I20260929 16:33:27.474785 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.474514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60852)
0929 16:33:27.474653 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.474715 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 16:33:27.475196 4639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } attempt: 2
I20260929 16:33:27.475572 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.475330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60856)
0929 16:33:27.475453 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.475512 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:33:27.475715 4714 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:42687: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
I20260929 16:33:27.475827 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.475564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60872)
0929 16:33:27.475687 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.475747 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:33:27.476604 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.476298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60886)
0929 16:33:27.476482 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.476535 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260929 16:33:27.476783 4639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:42687: Network error: Client connection negotiation failed: client connection to 127.23.44.60:42687: connect: Connection refused (error 111)
W20260929 16:33:27.480799 4727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:27.480813 4726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:27.480858 4720 server_base.cc:1034] running on GCE node
W20260929 16:33:27.480813 4729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:27.481556 4720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:27.482564 4720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:27.483745 4720 hybrid_clock.cc:648] HybridClock initialized: now 1790699607483716 us; error 41 us; skew 500 ppm
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:27.486373 4720 init.cc:377] Logged in from keytab as kudu/127.23.44.60@KRBTEST.COM (short username kudu)
I20260929 16:33:27.487232 4720 webserver.cc:460] Webserver started at http://127.23.44.60:33265/ using document root <none> and password file <none>
I20260929 16:33:27.487812 4720 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:27.487902 4720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:27.488122 4720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:27.490090 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/data/instance:
uuid: "161cb16c78f54b7ca7d79389471f0e60"
format_stamp: "Formatted at 2026-09-29 16:33:27 on dist-test-slave-mlmz"
server_key: "e20755ceca5934aaa8df9ff80d30e634"
server_key_iv: "6993b853a59517cc716b930618d29f08"
server_key_version: "encryptionkey@0"
I20260929 16:33:27.490628 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/wal/instance:
uuid: "161cb16c78f54b7ca7d79389471f0e60"
format_stamp: "Formatted at 2026-09-29 16:33:27 on dist-test-slave-mlmz"
server_key: "e20755ceca5934aaa8df9ff80d30e634"
server_key_iv: "6993b853a59517cc716b930618d29f08"
server_key_version: "encryptionkey@0"
I20260929 16:33:27.494351 4720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:33:27.496804 4736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:27.498143 4720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:33:27.498278 4720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/wal
uuid: "161cb16c78f54b7ca7d79389471f0e60"
format_stamp: "Formatted at 2026-09-29 16:33:27 on dist-test-slave-mlmz"
server_key: "e20755ceca5934aaa8df9ff80d30e634"
server_key_iv: "6993b853a59517cc716b930618d29f08"
server_key_version: "encryptionkey@0"
I20260929 16:33:27.498394 4720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:27.509786 4720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:27.512564 4720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:27.512748 4720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:27.519654 4720 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.60:42687
I20260929 16:33:27.519677 4788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.60:42687 every 8 connection(s)
I20260929 16:33:27.520509 4720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
I20260929 16:33:27.523058 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:27.526508 4789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:33:27.528358 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 4720
I20260929 16:33:27.528508 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699450427356-23728-0/minicluster-data/master-2/wal/instance
I20260929 16:33:27.528821 23728 external_mini_cluster.cc:1428] Setting key c82d7fe4e0731e8082f5b5d2271acc1e
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.60@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.60@KRBTEST.COM
I20260929 16:33:27.540032 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.528184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.23.44.62:39765)
0929 16:33:27.528349 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.528354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.529199 (+ 845us) server_negotiation.cc:407] Connection header received
0929 16:33:27.530317 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.530320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.530383 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.530461 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.531336 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.531809 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.532852 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.533153 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.533230 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.535912 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.535916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.535919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.535947 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.538168 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.538581 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.538584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.538585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.538655 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.539007 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.539010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.539012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.539182 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.539287 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.539759 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.539869 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:33:27.540773 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.529475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60888 (local address 127.23.44.60:42687)
0929 16:33:27.529817 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.529824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.529856 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:33:27.530138 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.530155 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.530323 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.530489 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.531197 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.532156 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.533006 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.533549 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.533713 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.536313 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.536321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.536333 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.536358 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.538168 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.538923 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.538928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.538929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.539000 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.539271 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.539275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.539277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.539599 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.539763 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.540065 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.540200 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":271,"thread_start_us":102,"threads_started":1}
I20260929 16:33:27.540772 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.528535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39765 (local address 127.0.0.1:54406)
0929 16:33:27.529037 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.529089 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:27.530128 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:27.530524 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:27.530531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:27.530925 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:27.531186 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.531197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.531917 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.531920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.532738 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.532746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.533320 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.533324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.533627 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.534397 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:27.534411 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:27.535747 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:27.538269 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.538276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.538282 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.538494 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.538751 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.538755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.538757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.538895 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.539285 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:27.539292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:27.539520 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 16:33:27.539691 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 16:33:27.539969 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":82,"spinlock_wait_cycles":26880,"thread_start_us":110,"threads_started":1}
I20260929 16:33:27.541908 4789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.60@KRBTEST.COM for kudu/127.23.44.61@KRBTEST.COM
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:27.550134 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.542355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:33069 (local address 127.0.0.1:47096)
0929 16:33:27.542507 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.542520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:27.542586 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:27.542839 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:27.542842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:27.542996 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:27.543106 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.543112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.543818 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.543825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.544321 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.544328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.544764 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.544767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.544874 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.545340 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:27.545352 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:27.546832 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:27.548616 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.548618 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.548620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.548880 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.549157 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.549161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.549164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.549275 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.549665 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:27.549668 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:27.549763 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 16:33:27.549877 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 16:33:27.549993 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 16:33:27.550166 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.542474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.23.44.61:33069)
0929 16:33:27.542585 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.542588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.542600 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:33:27.542667 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.542671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.542727 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.542810 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.543221 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.543711 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.544442 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.544659 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.544765 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.546959 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.546962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.546964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.546985 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.548510 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.548984 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.548986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.548987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.549063 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.549405 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.549407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.549408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.549562 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.549675 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.549928 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.550031 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:33:27.551064 4789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:33:27.551116 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.542895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.23.44.62:39765)
0929 16:33:27.543002 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.543006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.543018 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:33:27.543090 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.543093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.543127 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.543199 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.543802 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.544246 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.544996 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.545214 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.545367 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.547788 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.547792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.547794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.547817 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.549440 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.549817 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.549821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.549822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.549908 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.550163 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.550167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.550169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.550349 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.550542 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.550841 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.550972 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.60@KRBTEST.COM for kudu/127.23.44.60@KRBTEST.COM
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.61@KRBTEST.COM
I20260929 16:33:27.559615 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.551496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60892)
0929 16:33:27.551577 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.551591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:27.551658 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:27.551978 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:27.551981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:27.552151 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:27.552241 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.552248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.553232 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.553235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.554068 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.554076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.554621 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.554624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.554677 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.555134 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:27.555147 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:27.556513 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:27.558292 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.558298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.558300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.558474 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.558731 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.558736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.558738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.558811 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.559153 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:27.559156 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:27.559240 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 16:33:27.559342 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:33:27.559463 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 16:33:27.559636 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.551555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.23.44.60:42687)
0929 16:33:27.551732 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.551756 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.551772 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:33:27.551804 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.551809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.551863 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.551949 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.552361 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.553133 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.554166 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.554519 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.554618 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.556636 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.556638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.556640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.556659 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.558188 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.558563 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.558569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.558571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.558645 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.558907 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.558910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.558911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.559063 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.559162 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.559386 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.559513 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":124}
I20260929 16:33:27.560770 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.552259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.23.44.61:33069)
0929 16:33:27.552359 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.552363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.552383 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:33:27.552485 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.552487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.552516 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.552575 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.553228 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.553973 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.554947 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.555170 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.555273 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.557664 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.557669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.557672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.557700 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.559114 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.559522 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.559526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.559528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.559604 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.559918 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.559921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.559923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.560124 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.560295 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.560500 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.560616 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:33:27.563001 4789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60: Bootstrap starting.
I20260929 16:33:27.564124 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:33:27.565744 4789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:27.566610 4789 log.cc:824] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:27.566867 4714 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } attempt: 3
I20260929 16:33:27.569155 4789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60: No bootstrap required, opened a new log
I20260929 16:33:27.573696 4789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } }
I20260929 16:33:27.574043 4789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:27.574156 4789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161cb16c78f54b7ca7d79389471f0e60, State: Initialized, Role: FOLLOWER
I20260929 16:33:27.574721 4789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } }
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.61@KRBTEST.COM for kudu/127.23.44.60@KRBTEST.COM
I20260929 16:33:27.576129 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.566618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.23.44.62:39765)
0929 16:33:27.566733 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.566737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.567149 (+ 412us) server_negotiation.cc:407] Connection header received
0929 16:33:27.567320 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.567323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.567373 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.567469 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.568345 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.569111 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.570380 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.570713 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.570792 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.572514 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.572520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.572525 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.572553 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.574211 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.574732 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.574735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.574737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.574824 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.575162 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.575168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.575171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.575433 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.575537 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.575923 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.576028 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:33:27.576136 4789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:27.576259 4639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } attempt: 3
I20260929 16:33:27.577415 4803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } } }
I20260929 16:33:27.578918 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.566522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.23.44.61:33069)
0929 16:33:27.566809 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.566813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.567822 (+ 1009us) server_negotiation.cc:407] Connection header received
0929 16:33:27.567864 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.567869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.567921 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.567988 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.568991 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.569722 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.570870 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.571504 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.571628 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.573809 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.573813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.573815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.573840 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.576304 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.577369 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.577373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.577375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.577689 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.578036 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.578039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.578041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.578226 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.578340 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.578652 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.578757 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260929 16:33:27.579798 4803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:33:27.580375 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.567186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60908)
0929 16:33:27.567339 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.567352 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:27.567418 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:27.570574 (+ 3156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:27.570577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:27.570762 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:27.570868 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.570875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.572088 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.572092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.573004 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.573015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.573723 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.573727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.573829 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.574360 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:27.574376 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:27.575986 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:27.578588 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.578594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.578597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.578899 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.579306 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.579309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.579312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.579409 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.579892 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:27.579895 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:27.579979 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 16:33:27.580079 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:33:27.580202 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 16:33:27.581192 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.567647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.23.44.60:42687)
0929 16:33:27.570322 (+ 2675us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.570332 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.570347 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:33:27.570381 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.570384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.570423 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.570989 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.571094 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.571947 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.573163 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.573572 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.574045 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.576477 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.576481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.576484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.576517 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.578448 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.579023 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.579027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.579028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.579114 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.579503 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.579506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.579508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.579747 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.579834 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.580651 (+ 817us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.580768 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2607,"thread_start_us":87,"threads_started":1}
I20260929 16:33:27.584131 4714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282: Bootstrap starting.
I20260929 16:33:27.581105 4720 master_runner.cc:186] Error getting master registration for 127.23.44.62:39765: OK, instance_id { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" instance_seqno: 1790699607112628 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:33:27.584642 4720 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 16:33:27 dist-test-slave-mlmz krb5kdc[4558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.60@KRBTEST.COM
I20260929 16:33:27.586143 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.567123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.23.44.60:42687)
0929 16:33:27.567229 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.567234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.567332 (+ 98us) server_negotiation.cc:407] Connection header received
0929 16:33:27.567895 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.567898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.567936 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.567987 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.568932 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.569695 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.578417 (+ 8722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.578735 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.578805 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.582626 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.582633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.582638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.582667 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.584408 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.584870 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.584875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.584877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.584951 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.585279 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.585286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.585291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.585490 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.585598 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.585923 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.586021 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 16:33:27.586896 4815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:33:27.586957 4815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:33:27.587113 4714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:27.588063 4714 log.cc:824] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:27.589907 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.576568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:42687 (local address 127.0.0.1:60910)
0929 16:33:27.576728 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.576741 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:27.576817 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:27.580341 (+ 3524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:27.580344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:27.580508 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:27.580612 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.580619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.581708 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.581712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.582596 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:27.582606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.583820 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.583826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:27.583935 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.584374 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:27.584388 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:27.585998 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:27.588120 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.588124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.588126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.588383 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.588769 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:27.588773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:27.588775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:27.588900 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:27.589379 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:27.589383 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:27.589487 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 16:33:27.589612 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 16:33:27.589745 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 16:33:27.590845 4714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282: No bootstrap required, opened a new log
I20260929 16:33:27.591059 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.576598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.23.44.60:42687)
0929 16:33:27.580113 (+ 3515us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:27.580124 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:27.580136 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:33:27.580169 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:27.580173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:27.580210 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:27.580715 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:27.580808 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.581585 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.582723 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:27.583412 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:27.584926 (+ 1514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:27.586157 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:27.586161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:27.586164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:27.586188 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:27.587978 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.588531 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.588534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.588536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.588645 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:27.589034 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:27.589039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:27.589041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:27.589207 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:27.589387 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:27.590706 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0929 16:33:27.590872 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3161,"thread_start_us":86,"threads_started":1}
I20260929 16:33:27.593894 4639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2: Bootstrap starting.
I20260929 16:33:27.594913 4714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } }
I20260929 16:33:27.595134 4714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:27.595199 4714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79504bb242d64d698415415b882e6282, State: Initialized, Role: FOLLOWER
I20260929 16:33:27.595615 4714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } }
I20260929 16:33:27.596021 4818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } } }
I20260929 16:33:27.596182 4818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:33:27.596274 4639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:27.596841 4714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:27.597043 4639 log.cc:824] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:27.599058 4639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2: No bootstrap required, opened a new log
I20260929 16:33:27.602864 4639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } }
I20260929 16:33:27.603114 4639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:27.603204 4639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 612c059780bd409eaa96898c59eac9b2, State: Initialized, Role: FOLLOWER
I20260929 16:33:27.603669 4639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } }
I20260929 16:33:27.599990 4646 master_runner.cc:186] Error getting master registration for 127.23.44.62:39765: OK, instance_id { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" instance_seqno: 1790699607112628 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:33:27.603888 4646 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:33:27.604322 4832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } } }
W20260929 16:33:27.604419 4830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 16:33:27.604452 4832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 16:33:27.604486 4830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:33:27.605228 4639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:33:27.609053 4592 tablet.cc:2369] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:33:27.611310 4843 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:33:27.611442 4843 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:33:27.612830 4571 master_runner.cc:263] No leader master found from master 612c059780bd409eaa96898c59eac9b2
I20260929 16:33:27.612901 4571 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:33:27.654963 4818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:27.655170 4818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } }
I20260929 16:33:27.656222 4818 leader_election.cc:288] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 612c059780bd409eaa96898c59eac9b2 (127.23.44.62:39765), 161cb16c78f54b7ca7d79389471f0e60 (127.23.44.60:42687)
I20260929 16:33:27.657107 4613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79504bb242d64d698415415b882e6282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "612c059780bd409eaa96898c59eac9b2" is_pre_election: true
I20260929 16:33:27.657455 4613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79504bb242d64d698415415b882e6282 in term 0.
I20260929 16:33:27.657869 4763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79504bb242d64d698415415b882e6282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "161cb16c78f54b7ca7d79389471f0e60" is_pre_election: true
I20260929 16:33:27.658150 4763 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79504bb242d64d698415415b882e6282 in term 0.
I20260929 16:33:27.658558 4663 leader_election.cc:302] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161cb16c78f54b7ca7d79389471f0e60, 79504bb242d64d698415415b882e6282; no voters:
I20260929 16:33:27.659152 4818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:33:27.659240 4818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:33:27.659292 4818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:27.660604 4818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } }
I20260929 16:33:27.660931 4818 leader_election.cc:288] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [CANDIDATE]: Term 1 election: Requested vote from peers 612c059780bd409eaa96898c59eac9b2 (127.23.44.62:39765), 161cb16c78f54b7ca7d79389471f0e60 (127.23.44.60:42687)
I20260929 16:33:27.661576 4763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79504bb242d64d698415415b882e6282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "161cb16c78f54b7ca7d79389471f0e60"
I20260929 16:33:27.661695 4763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:27.662925 4763 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79504bb242d64d698415415b882e6282 in term 1.
I20260929 16:33:27.663216 4663 leader_election.cc:302] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161cb16c78f54b7ca7d79389471f0e60, 79504bb242d64d698415415b882e6282; no voters:
I20260929 16:33:27.663455 4613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79504bb242d64d698415415b882e6282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "612c059780bd409eaa96898c59eac9b2"
I20260929 16:33:27.663573 4613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:27.664780 4613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79504bb242d64d698415415b882e6282 in term 1.
I20260929 16:33:27.665203 4818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:27.665484 4818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [term 1 LEADER]: Becoming Leader. State: Replica: 79504bb242d64d698415415b882e6282, State: Running, Role: LEADER
I20260929 16:33:27.665814 4818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } }
I20260929 16:33:27.667906 4818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79504bb242d64d698415415b882e6282. Latest consensus state: current_term: 1 leader_uuid: "79504bb242d64d698415415b882e6282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } } }
I20260929 16:33:27.668015 4818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:27.668375 4847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:27.669324 4847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:27.675674 4763 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [term 1 FOLLOWER]: Refusing update from remote peer 79504bb242d64d698415415b882e6282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:27.676266 4818 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:27.680125 4803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79504bb242d64d698415415b882e6282. Latest consensus state: current_term: 1 leader_uuid: "79504bb242d64d698415415b882e6282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } } }
I20260929 16:33:27.680235 4803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:33:27.681046 4613 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [term 1 FOLLOWER]: Refusing update from remote peer 79504bb242d64d698415415b882e6282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:27.681581 4818 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:27.685523 4832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79504bb242d64d698415415b882e6282. Latest consensus state: current_term: 1 leader_uuid: "79504bb242d64d698415415b882e6282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } } }
I20260929 16:33:27.685643 4832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:33:27.690379 4845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79504bb242d64d698415415b882e6282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } } }
I20260929 16:33:27.690501 4845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:27.691533 4832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79504bb242d64d698415415b882e6282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } } }
I20260929 16:33:27.691638 4832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:33:27.692145 4848 mvcc.cc:205] Tried to move back new op lower bound from 7334705593033060352 to 7334705593002389504. Current Snapshot: MvccSnapshot[applied={T|T < 7334705593033060352}]
I20260929 16:33:27.693943 4851 mvcc.cc:205] Tried to move back new op lower bound from 7334705593033060352 to 7334705593002389504. Current Snapshot: MvccSnapshot[applied={T|T < 7334705593033060352}]
I20260929 16:33:27.694639 4818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79504bb242d64d698415415b882e6282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } } }
I20260929 16:33:27.694744 4818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:27.695655 4803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79504bb242d64d698415415b882e6282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "612c059780bd409eaa96898c59eac9b2" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39765 } } peers { permanent_uuid: "79504bb242d64d698415415b882e6282" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 33069 } } peers { permanent_uuid: "161cb16c78f54b7ca7d79389471f0e60" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 42687 } } }
I20260929 16:33:27.695770 4803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:33:27.696218 4847 catalog_manager.cc:1347] Generated new cluster ID: 835067feec8c41d68bb9cf56c947b6f1
I20260929 16:33:27.696280 4847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:27.708683 4849 mvcc.cc:205] Tried to move back new op lower bound from 7334705593033060352 to 7334705593002389504. Current Snapshot: MvccSnapshot[applied={T|T < 7334705593033060352}]
I20260929 16:33:27.738521 4847 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:33:27.739683 4847 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:27.750618 4847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79504bb242d64d698415415b882e6282: Generated new TSK 0
I20260929 16:33:27.751360 4847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:33:28.588186 4815 catalog_manager.cc:1259] Loaded cluster ID: 835067feec8c41d68bb9cf56c947b6f1
I20260929 16:33:28.588279 4815 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60: loading cluster ID for follower catalog manager: success
I20260929 16:33:28.589972 4815 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60: acquiring CA information for follower catalog manager: success
I20260929 16:33:28.591131 4815 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 161cb16c78f54b7ca7d79389471f0e60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:33:28.612561 4843 catalog_manager.cc:1259] Loaded cluster ID: 835067feec8c41d68bb9cf56c947b6f1
I20260929 16:33:28.612630 4843 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2: loading cluster ID for follower catalog manager: success
I20260929 16:33:28.614317 4843 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2: acquiring CA information for follower catalog manager: success
I20260929 16:33:28.615370 4843 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 612c059780bd409eaa96898c59eac9b2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:33:28.817168 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 4571
I20260929 16:33:28.823231 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 4646
I20260929 16:33:28.829144 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 4720
2026-09-29T16:33:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3039 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:33:28 dist-test-slave-mlmz krb5kdc[4862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:33:28 dist-test-slave-mlmz krb5kdc[4862](info): set up 2 sockets
Sep 29 16:33:28 dist-test-slave-mlmz krb5kdc[4862](info): commencing operation
krb5kdc: starting...
W20260929 16:33:29.853772 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:33:29 dist-test-slave-mlmz krb5kdc[4862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:33:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:33:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:33:29.974103 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:46841
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:46841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:30.077783 4875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:30.078001 4875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:30.078037 4875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:30.081919 4875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:30.081965 4875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:30.081984 4875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:30.082000 4875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:30.082015 4875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:30.086915 4875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:46841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:46841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:30.087775 4875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:30.088726 4875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:30.096669 4880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:30.096601 4881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:30.096601 4883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:30.096745 4875 server_base.cc:1034] running on GCE node
I20260929 16:33:30.097409 4875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:30.098352 4875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:30.099558 4875 hybrid_clock.cc:648] HybridClock initialized: now 1790699610099521 us; error 50 us; skew 500 ppm
Sep 29 16:33:30 dist-test-slave-mlmz krb5kdc[4862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699610, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:30.102461 4875 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:33:30.103191 4875 webserver.cc:460] Webserver started at http://127.23.44.62:43433/ using document root <none> and password file <none>
I20260929 16:33:30.103662 4875 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:30.103714 4875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:30.103858 4875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:30.105736 4875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "bfa99af365a0494aba85e8a699c25d70"
format_stamp: "Formatted at 2026-09-29 16:33:30 on dist-test-slave-mlmz"
server_key: "d93c85da9b5b1edfac7aeb3f31d7a30c"
server_key_iv: "42a24d5819707e6455106a1579296dcd"
server_key_version: "encryptionkey@0"
I20260929 16:33:30.106204 4875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "bfa99af365a0494aba85e8a699c25d70"
format_stamp: "Formatted at 2026-09-29 16:33:30 on dist-test-slave-mlmz"
server_key: "d93c85da9b5b1edfac7aeb3f31d7a30c"
server_key_iv: "42a24d5819707e6455106a1579296dcd"
server_key_version: "encryptionkey@0"
I20260929 16:33:30.109603 4875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:33:30.111739 4890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:30.113036 4875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:33:30.113166 4875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "bfa99af365a0494aba85e8a699c25d70"
format_stamp: "Formatted at 2026-09-29 16:33:30 on dist-test-slave-mlmz"
server_key: "d93c85da9b5b1edfac7aeb3f31d7a30c"
server_key_iv: "42a24d5819707e6455106a1579296dcd"
server_key_version: "encryptionkey@0"
I20260929 16:33:30.113291 4875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:30.138052 4875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:30.140985 4875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:30.141175 4875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:30.147926 4875 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:46841
I20260929 16:33:30.147966 4942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:46841 every 8 connection(s)
I20260929 16:33:30.148788 4875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:30.149322 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 4875
I20260929 16:33:30.149454 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:33:30.149751 23728 external_mini_cluster.cc:1428] Setting key f316aff0b17134f58650c1151bfd8926
I20260929 16:33:30.151615 4943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:30.157750 4943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70: Bootstrap starting.
Sep 29 16:33:30 dist-test-slave-mlmz krb5kdc[4862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:30.160132 4943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:30.160961 4943 log.cc:824] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:30.163062 4943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70: No bootstrap required, opened a new log
I20260929 16:33:30.163848 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:30.150930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.23.44.62:46841)
0929 16:33:30.151329 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:30.151344 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:30.151376 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:33:30.152023 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:30.152040 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:30.152347 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:30.152600 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:30.153456 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:30.154426 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:30.155509 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:30.155941 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:30.156053 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:30.158873 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:30.158879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:30.158893 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:30.158921 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:30.160890 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:30.161422 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:30.161426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:30.161430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:30.161529 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:30.161874 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:30.161877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:30.161879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:30.162297 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:30.162491 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:30.162717 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 16:33:30.162977 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":136,"threads_started":1}
I20260929 16:33:30.166721 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:33:30.167032 4943 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa99af365a0494aba85e8a699c25d70" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46841 } }
I20260929 16:33:30.167284 4943 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:30.167333 4943 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa99af365a0494aba85e8a699c25d70, State: Initialized, Role: FOLLOWER
I20260929 16:33:30.167724 4943 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa99af365a0494aba85e8a699c25d70" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46841 } }
I20260929 16:33:30.167886 4943 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:30.167953 4943 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:30.168041 4943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:30.168897 4943 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa99af365a0494aba85e8a699c25d70" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46841 } }
I20260929 16:33:30.169157 4943 leader_election.cc:302] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfa99af365a0494aba85e8a699c25d70; no voters:
I20260929 16:33:30.169456 4943 leader_election.cc:288] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:33:30.169562 4948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:30.169832 4948 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [term 1 LEADER]: Becoming Leader. State: Replica: bfa99af365a0494aba85e8a699c25d70, State: Running, Role: LEADER
I20260929 16:33:30.170123 4948 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa99af365a0494aba85e8a699c25d70" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46841 } }
I20260929 16:33:30.170533 4943 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:30.171756 4950 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfa99af365a0494aba85e8a699c25d70. Latest consensus state: current_term: 1 leader_uuid: "bfa99af365a0494aba85e8a699c25d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa99af365a0494aba85e8a699c25d70" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46841 } } }
I20260929 16:33:30.171801 4949 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfa99af365a0494aba85e8a699c25d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa99af365a0494aba85e8a699c25d70" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46841 } } }
I20260929 16:33:30.171897 4949 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:30.171898 4950 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70 [sys.catalog]: This master's current role is: LEADER
W20260929 16:33:30.174988 4963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:33:30.175056 4963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:33:30.175141 4958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:30.175869 4958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:30.180802 4958 catalog_manager.cc:1347] Generated new cluster ID: 0cbbe4318e364d49b60855d4c47995d2
I20260929 16:33:30.180878 4958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:30.189518 4958 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:33:30.190325 4958 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:30.194072 4958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfa99af365a0494aba85e8a699c25d70: Generated new TSK 0
I20260929 16:33:30.194665 4958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:33:30.222523 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 4875
2026-09-29T16:33:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1391 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:33:30 dist-test-slave-mlmz krb5kdc[4970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:33:30 dist-test-slave-mlmz krb5kdc[4970](info): set up 2 sockets
Sep 29 16:33:30 dist-test-slave-mlmz krb5kdc[4970](info): commencing operation
krb5kdc: starting...
W20260929 16:33:31.261466 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:33:31 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:33:31.379951 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39531
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:39531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:31.484274 4983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:31.484506 4983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:31.484544 4983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:31.488473 4983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:31.488524 4983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:31.488543 4983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:31.488559 4983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:31.488574 4983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:31.493574 4983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:39531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:31.494532 4983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:31.495458 4983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:31.503376 4988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:31.503438 4989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:31.503414 4991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:31.503747 4983 server_base.cc:1034] running on GCE node
I20260929 16:33:31.504199 4983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:31.505177 4983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:31.506356 4983 hybrid_clock.cc:648] HybridClock initialized: now 1790699611506339 us; error 29 us; skew 500 ppm
Sep 29 16:33:31 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699611, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:31.514212 4983 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:33:31.515050 4983 webserver.cc:460] Webserver started at http://127.23.44.62:38287/ using document root <none> and password file <none>
I20260929 16:33:31.515630 4983 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:31.515689 4983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:31.515848 4983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:31.517892 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "2fda6a5dcfd740abb37c415a2d112ccd"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "1b3f841a91e9709bac7c589c4015b1b3"
server_key_iv: "ef6efa77e1a92c1c7eccd0883be5e6a4"
server_key_version: "encryptionkey@0"
I20260929 16:33:31.518380 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "2fda6a5dcfd740abb37c415a2d112ccd"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "1b3f841a91e9709bac7c589c4015b1b3"
server_key_iv: "ef6efa77e1a92c1c7eccd0883be5e6a4"
server_key_version: "encryptionkey@0"
I20260929 16:33:31.521786 4983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:33:31.523962 4998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:31.525209 4983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:33:31.525408 4983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "2fda6a5dcfd740abb37c415a2d112ccd"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "1b3f841a91e9709bac7c589c4015b1b3"
server_key_iv: "ef6efa77e1a92c1c7eccd0883be5e6a4"
server_key_version: "encryptionkey@0"
I20260929 16:33:31.525517 4983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:31.540411 4983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:31.543151 4983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:31.543341 4983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:31.550035 4983 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:39531
I20260929 16:33:31.550299 5050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:39531 every 8 connection(s)
I20260929 16:33:31.551007 4983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:31.553568 5051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:31.555212 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 4983
I20260929 16:33:31.555377 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:33:31.555660 23728 external_mini_cluster.cc:1428] Setting key 3115ae30bbc35ab1865672b66a3f9b99
I20260929 16:33:31.559924 5051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Bootstrap starting.
I20260929 16:33:31.562351 5051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:31.563207 5051 log.cc:824] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:33:31 dist-test-slave-mlmz krb5kdc[4970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:31.564953 5051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: No bootstrap required, opened a new log
I20260929 16:33:31.569476 5051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } }
I20260929 16:33:31.569710 5051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:31.569744 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:31.556942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42542 (local address 127.23.44.62:39531)
0929 16:33:31.557362 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:31.557381 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:31.557420 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:33:31.558138 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:31.558159 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:31.558549 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:31.558891 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:31.559520 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:31.560926 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:31.561802 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:31.562480 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:31.562648 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:31.564986 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:31.564994 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:31.565009 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:31.565037 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:31.566939 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:31.567352 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:31.567359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:31.567364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:31.567494 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:31.567750 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:31.567753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:31.567754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:31.568197 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:31.568397 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:31.568580 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 16:33:31.568857 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20260929 16:33:31.569787 5051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fda6a5dcfd740abb37c415a2d112ccd, State: Initialized, Role: FOLLOWER
I20260929 16:33:31.570216 5051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } }
I20260929 16:33:31.570333 5051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:31.570416 5051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:31.570531 5051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:31.571311 5051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } }
I20260929 16:33:31.571625 5051 leader_election.cc:302] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fda6a5dcfd740abb37c415a2d112ccd; no voters:
I20260929 16:33:31.571909 5051 leader_election.cc:288] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:33:31.571977 5056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:31.572235 5056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 1 LEADER]: Becoming Leader. State: Replica: 2fda6a5dcfd740abb37c415a2d112ccd, State: Running, Role: LEADER
I20260929 16:33:31.572504 5056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } }
I20260929 16:33:31.573048 5051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:31.573944 5057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } } }
I20260929 16:33:31.574074 5057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:31.574460 5065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:31.574409 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fda6a5dcfd740abb37c415a2d112ccd. Latest consensus state: current_term: 1 leader_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } } }
I20260929 16:33:31.574520 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:31.577310 5065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:31.582541 5065 catalog_manager.cc:1347] Generated new cluster ID: ca6e82918fe24d70b74abe213682c0ab
I20260929 16:33:31.582623 5065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:31.597159 5065 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:33:31.598045 5065 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:31.603022 5065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Generated new TSK 0
I20260929 16:33:31.603673 5065 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:33:31.634281 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39531
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:31.741214 5077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:31.741492 5077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:31.741564 5077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:31.745479 5077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:31.745558 5077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:31.745651 5077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:33:31.750633 5077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:31.751574 5077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:31.752516 5077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:31.761727 5082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:31.761770 5077 server_base.cc:1034] running on GCE node
W20260929 16:33:31.761670 5083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:31.761669 5085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:31.762306 5077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:31.762882 5077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:31.764050 5077 hybrid_clock.cc:648] HybridClock initialized: now 1790699611764016 us; error 42 us; skew 500 ppm
Sep 29 16:33:31 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699611, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:31.766772 5077 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:33:31.767565 5077 webserver.cc:460] Webserver started at http://127.23.44.1:46373/ using document root <none> and password file <none>
I20260929 16:33:31.768082 5077 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:31.768172 5077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:31.768392 5077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:31.770341 5077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "9647f9b96e8f4777853e35001d0513d4"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "dee3a703f00e1f5e0e8a91f2b55be63e"
server_key_iv: "0b441c3f67af8c9b0528860d4b2cc0d0"
server_key_version: "encryptionkey@0"
I20260929 16:33:31.770881 5077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "9647f9b96e8f4777853e35001d0513d4"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "dee3a703f00e1f5e0e8a91f2b55be63e"
server_key_iv: "0b441c3f67af8c9b0528860d4b2cc0d0"
server_key_version: "encryptionkey@0"
I20260929 16:33:31.774471 5077 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 16:33:31.776793 5092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:31.777989 5077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:33:31.778122 5077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "9647f9b96e8f4777853e35001d0513d4"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "dee3a703f00e1f5e0e8a91f2b55be63e"
server_key_iv: "0b441c3f67af8c9b0528860d4b2cc0d0"
server_key_version: "encryptionkey@0"
I20260929 16:33:31.778254 5077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:31.788479 5077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:31.791322 5077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:31.791518 5077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:31.792052 5077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:31.792809 5077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:31.792877 5077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:31.792941 5077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:31.792989 5077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:31.801491 5077 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:36121
I20260929 16:33:31.801527 5205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:36121 every 8 connection(s)
I20260929 16:33:31.802279 5077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:31.809803 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 5077
I20260929 16:33:31.809939 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:33:31.810318 23728 external_mini_cluster.cc:1428] Setting key f4c98d29da24357424a0bbd89f71cc14
Sep 29 16:33:31 dist-test-slave-mlmz krb5kdc[4970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699611, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:31.815517 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:31.803962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:37643 (local address 127.23.44.62:39531)
0929 16:33:31.804088 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:31.804091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:31.804860 (+ 769us) server_negotiation.cc:407] Connection header received
0929 16:33:31.806286 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:31.806288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:31.806345 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:31.806449 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:31.807499 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:31.808029 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:31.809122 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:31.809435 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:31.809545 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:31.812010 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:31.812014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:31.812016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:31.812042 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:31.813491 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:31.813970 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:31.813975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:31.813977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:31.814056 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:31.814409 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:31.814412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:31.814430 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:31.814614 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:31.814695 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:31.815303 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 16:33:31.815404 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:33:31.816568 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:31.804313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:37643)
0929 16:33:31.804728 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:31.804766 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:31.806082 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:31.806609 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:31.806615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:31.806941 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:31.807362 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:31.807375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:31.808152 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:31.808158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:31.808992 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:31.809001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:31.809574 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:31.809579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:31.809758 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:31.810339 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:31.810354 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:31.811845 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:31.813602 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:31.813607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:31.813620 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:31.813869 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:31.814147 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:31.814151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:31.814153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:31.814292 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:31.814722 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:31.814731 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:31.815026 (+ 295us) client_negotiation.cc:769] Sending connection context
0929 16:33:31.815233 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 16:33:31.815566 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260929 16:33:31.817871 5206 heartbeater.cc:347] Connected to a master server at 127.23.44.62:39531
I20260929 16:33:31.818113 5206 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:31.818661 5206 heartbeater.cc:511] Master 127.23.44.62:39531 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
I20260929 16:33:31.820186 5015 ts_manager.cc:194] Registered new tserver with Master: 9647f9b96e8f4777853e35001d0513d4 (127.23.44.1:36121)
I20260929 16:33:31.821408 5015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:37643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:33:31.838482 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39531
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:31.944007 5211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:31.944222 5211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:31.944260 5211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:31.948040 5211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:31.948084 5211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:31.948137 5211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:33:31.952894 5211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:31.953763 5211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:31.954589 5211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:31.963428 5219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:31.963435 5217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:31.963662 5211 server_base.cc:1034] running on GCE node
W20260929 16:33:31.963689 5216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:31.964092 5211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:31.964676 5211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:31.965886 5211 hybrid_clock.cc:648] HybridClock initialized: now 1790699611965868 us; error 23 us; skew 500 ppm
Sep 29 16:33:31 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699611, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:31.968518 5211 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:33:31.969367 5211 webserver.cc:460] Webserver started at http://127.23.44.2:42307/ using document root <none> and password file <none>
I20260929 16:33:31.969830 5211 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:31.969884 5211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:31.970029 5211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:31.971820 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "a2a607c1976042419803ee59b3f91cec"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "c82c547459ed876f19ce3fe90c78a4af"
server_key_iv: "050be821fd4c6d02ae106f06d3f4295a"
server_key_version: "encryptionkey@0"
I20260929 16:33:31.972311 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "a2a607c1976042419803ee59b3f91cec"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "c82c547459ed876f19ce3fe90c78a4af"
server_key_iv: "050be821fd4c6d02ae106f06d3f4295a"
server_key_version: "encryptionkey@0"
I20260929 16:33:31.976037 5211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:33:31.978333 5226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:31.979523 5211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:33:31.979612 5211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "a2a607c1976042419803ee59b3f91cec"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "c82c547459ed876f19ce3fe90c78a4af"
server_key_iv: "050be821fd4c6d02ae106f06d3f4295a"
server_key_version: "encryptionkey@0"
I20260929 16:33:31.979691 5211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:32.017042 5211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:32.020277 5211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:32.020421 5211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:32.021136 5211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:32.021991 5211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:32.022037 5211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:32.022076 5211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:32.022092 5211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:32.030923 5211 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:39867
I20260929 16:33:32.030949 5339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:39867 every 8 connection(s)
I20260929 16:33:32.031733 5211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:32.034513 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 5211
I20260929 16:33:32.034652 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:33:32.035000 23728 external_mini_cluster.cc:1428] Setting key e2067e5e73c7ad4533e415c326528e85
Sep 29 16:33:32 dist-test-slave-mlmz krb5kdc[4970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699611, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:33:32.046013 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.033363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:59185 (local address 127.23.44.62:39531)
0929 16:33:32.033502 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:32.033507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:32.034278 (+ 771us) server_negotiation.cc:407] Connection header received
0929 16:33:32.035891 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:32.035894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:32.035940 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:32.036029 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:32.037414 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.037944 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.039063 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.039294 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.039406 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.042181 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:32.042185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:32.042190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:32.042224 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:32.043884 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:32.044396 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:32.044403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:32.044406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:32.044495 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:32.044812 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:32.044816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:32.044817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:32.045077 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:32.045203 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:32.045714 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 16:33:32.045852 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:33:32.046801 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.033749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:59185)
0929 16:33:32.034140 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.034178 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:32.035708 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:32.036168 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:32.036175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:32.036587 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:32.037232 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.037292 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.038069 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.038072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.038932 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.038941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.039441 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.039446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.039670 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.040316 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:32.040340 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:32.042018 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:32.043999 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:32.044003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:32.044012 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:32.044272 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:32.044591 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:32.044593 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:32.044594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:32.044711 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:32.045196 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:32.045200 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:32.045481 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 16:33:32.045654 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 16:33:32.045946 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
I20260929 16:33:32.047992 5340 heartbeater.cc:347] Connected to a master server at 127.23.44.62:39531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
I20260929 16:33:32.048272 5340 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:32.048807 5340 heartbeater.cc:511] Master 127.23.44.62:39531 requested a full tablet report, sending...
I20260929 16:33:32.049899 5015 ts_manager.cc:194] Registered new tserver with Master: a2a607c1976042419803ee59b3f91cec (127.23.44.2:39867)
I20260929 16:33:32.050474 5015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:59185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:33:32.063688 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39531
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:32.167779 5345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:32.167994 5345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:32.168100 5345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:32.172029 5345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:32.172078 5345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:32.172161 5345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:33:32.177331 5345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:32.178242 5345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:32.179157 5345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:32.188385 5350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:32.188365 5351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:32.188365 5353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:32.188781 5345 server_base.cc:1034] running on GCE node
I20260929 16:33:32.189116 5345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:32.189741 5345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:32.190901 5345 hybrid_clock.cc:648] HybridClock initialized: now 1790699612190887 us; error 25 us; skew 500 ppm
Sep 29 16:33:32 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:32.193572 5345 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:33:32.194329 5345 webserver.cc:460] Webserver started at http://127.23.44.3:43041/ using document root <none> and password file <none>
I20260929 16:33:32.194849 5345 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:32.194933 5345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:32.195153 5345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:32.197017 5345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "8fe05f378ccb40309263fec338aa1665"
format_stamp: "Formatted at 2026-09-29 16:33:32 on dist-test-slave-mlmz"
server_key: "aecdedbdd115f3738d76575ef5adae74"
server_key_iv: "9c27520be66f8c6710f1d0244bba6f4d"
server_key_version: "encryptionkey@0"
I20260929 16:33:32.197570 5345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "8fe05f378ccb40309263fec338aa1665"
format_stamp: "Formatted at 2026-09-29 16:33:32 on dist-test-slave-mlmz"
server_key: "aecdedbdd115f3738d76575ef5adae74"
server_key_iv: "9c27520be66f8c6710f1d0244bba6f4d"
server_key_version: "encryptionkey@0"
I20260929 16:33:32.200947 5345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 16:33:32.203265 5360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:32.204404 5345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:33:32.204542 5345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "8fe05f378ccb40309263fec338aa1665"
format_stamp: "Formatted at 2026-09-29 16:33:32 on dist-test-slave-mlmz"
server_key: "aecdedbdd115f3738d76575ef5adae74"
server_key_iv: "9c27520be66f8c6710f1d0244bba6f4d"
server_key_version: "encryptionkey@0"
I20260929 16:33:32.204644 5345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:32.215181 5345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:32.218180 5345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:32.218369 5345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:32.218933 5345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:32.219671 5345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:32.219738 5345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:32.219803 5345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:32.219846 5345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:32.228127 5345 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:44141
I20260929 16:33:32.228158 5473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:44141 every 8 connection(s)
I20260929 16:33:32.229064 5345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
Sep 29 16:33:32 dist-test-slave-mlmz krb5kdc[4970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:32.239274 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 5345
I20260929 16:33:32.239399 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:33:32.239732 23728 external_mini_cluster.cc:1428] Setting key 84e7c797fb3fd959a75c7d74df87845e
I20260929 16:33:32.242321 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.230961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:57591 (local address 127.23.44.62:39531)
0929 16:33:32.231103 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:32.231107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:32.231847 (+ 740us) server_negotiation.cc:407] Connection header received
0929 16:33:32.233113 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:32.233115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:32.233162 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:32.233236 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:32.234381 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.234898 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.235973 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.236195 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.236282 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.238513 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:32.238516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:32.238518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:32.238541 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:32.240173 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:32.240743 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:32.240747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:32.240749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:32.240834 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:32.241208 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:32.241210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:32.241212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:32.241465 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:32.241547 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:32.242076 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 16:33:32.242171 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:33:32.243001 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.231224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:57591)
0929 16:33:32.231681 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.231720 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:32.232914 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:32.233476 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:32.233482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:32.233817 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:32.234245 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.234257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.235002 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.235004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.235855 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.235863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.236333 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.236336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.236464 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.236977 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:32.236992 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:32.238370 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:32.240315 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:32.240320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:32.240333 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:32.240642 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:32.240928 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:32.240932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:32.240933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:32.241111 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:32.241571 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:32.241577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:32.241837 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 16:33:32.241985 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 16:33:32.242207 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":108,"threads_started":1}
I20260929 16:33:32.244098 5474 heartbeater.cc:347] Connected to a master server at 127.23.44.62:39531
I20260929 16:33:32.244338 5474 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:32.244792 5474 heartbeater.cc:511] Master 127.23.44.62:39531 requested a full tablet report, sending...
I20260929 16:33:32.245776 5015 ts_manager.cc:194] Registered new tserver with Master: 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141)
I20260929 16:33:32.246322 5015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:57591
I20260929 16:33:32.253438 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:33:32.261863 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.255264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.23.44.62:39531)
0929 16:33:32.255360 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:32.255363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:32.255593 (+ 230us) server_negotiation.cc:407] Connection header received
0929 16:33:32.255678 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:32.255680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:32.255715 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:32.255775 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:32.256375 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.256864 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.257694 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.257902 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.257978 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.259209 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:32.259211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:32.259213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:32.259229 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:32.260381 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:32.260730 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:32.260733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:32.260734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:32.260799 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:32.261022 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:32.261024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:32.261025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:32.261203 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:32.261372 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:32.261575 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 16:33:32.261658 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 16:33:32.264269 5015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:33:32.267160 5015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:33:32.281004 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.275214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:39867 (local address 127.0.0.1:39638)
0929 16:33:32.275464 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.275477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:32.275559 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:32.276659 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:32.276662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:32.276681 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:32.276805 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.276812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.278122 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.278126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.279423 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.279433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.280534 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.280537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.280659 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.280681 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:33:32.280745 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:33:32.280891 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":183,"thread_start_us":107,"threads_started":1}
I20260929 16:33:32.281046 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.276070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.23.44.1:36121)
0929 16:33:32.276369 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:32.276383 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:32.276405 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:33:32.276471 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:32.276479 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:32.276664 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:32.277047 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:32.277160 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.277942 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.279816 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.280352 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.280474 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.280610 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:32.280788 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 16:33:32.280884 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260929 16:33:32.281035 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.275386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:36121 (local address 127.0.0.1:43950)
0929 16:33:32.275615 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.275630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:32.275715 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:32.276824 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:32.276827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:32.276847 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:32.276941 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.276945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.278098 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.278101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.279705 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.279718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.280473 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.280477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.280577 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.280612 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:33:32.280754 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 16:33:32.280890 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":121,"thread_start_us":103,"threads_started":1}
I20260929 16:33:32.281684 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.274702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:44141 (local address 127.0.0.1:33752)
0929 16:33:32.275030 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.275079 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:32.275200 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:32.276447 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:32.276452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:32.276474 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:32.276615 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.276623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.278059 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.278064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.279570 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.279580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.280549 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.280552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.281373 (+ 821us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.281396 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:33:32.281482 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:33:32.281544 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":795,"thread_start_us":78,"threads_started":1}
I20260929 16:33:32.282100 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.274799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.23.44.3:44141)
0929 16:33:32.275961 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:32.275975 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:32.275999 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:33:32.276071 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:32.276076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:32.276272 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:32.276440 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:32.276740 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.277920 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.279690 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.280436 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.280640 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.280776 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:32.281863 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0929 16:33:32.281953 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":87,"threads_started":1}
I20260929 16:33:32.282378 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.275701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39638 (local address 127.23.44.2:39867)
0929 16:33:32.275993 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:32.276006 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:32.276027 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:33:32.276099 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:32.276104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:32.276472 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:32.276660 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:32.276895 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.278005 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.279548 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.280380 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.281887 (+ 1507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.282061 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:32.282112 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 16:33:32.282214 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":122,"threads_started":1}
I20260929 16:33:32.283905 5140 tablet_service.cc:1467] Processing CreateTablet for tablet 99ec1a22878a45b9a44e34d93b38570b (DEFAULT_TABLE table=test-table [id=050a314099ce4378b3257c49e06a9131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:32.284839 5140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ec1a22878a45b9a44e34d93b38570b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:32.284664 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 99ec1a22878a45b9a44e34d93b38570b (DEFAULT_TABLE table=test-table [id=050a314099ce4378b3257c49e06a9131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:32.285629 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ec1a22878a45b9a44e34d93b38570b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:32.291240 5491 tablet_bootstrap.cc:492] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Bootstrap starting.
I20260929 16:33:32.292105 5492 tablet_bootstrap.cc:492] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Bootstrap starting.
I20260929 16:33:32.291899 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 99ec1a22878a45b9a44e34d93b38570b (DEFAULT_TABLE table=test-table [id=050a314099ce4378b3257c49e06a9131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:32.292889 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ec1a22878a45b9a44e34d93b38570b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:32.293373 5491 tablet_bootstrap.cc:654] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:32.294397 5491 log.cc:824] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:32.294539 5492 tablet_bootstrap.cc:654] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:32.295331 5492 log.cc:824] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:32.296315 5491 tablet_bootstrap.cc:492] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: No bootstrap required, opened a new log
I20260929 16:33:32.296548 5491 ts_tablet_manager.cc:1397] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260929 16:33:32.297108 5492 tablet_bootstrap.cc:492] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: No bootstrap required, opened a new log
I20260929 16:33:32.297308 5492 ts_tablet_manager.cc:1397] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:33:32.299345 5491 raft_consensus.cc:348] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.299693 5491 raft_consensus.cc:374] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:32.299716 5495 tablet_bootstrap.cc:492] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665: Bootstrap starting.
I20260929 16:33:32.299798 5491 raft_consensus.cc:729] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9647f9b96e8f4777853e35001d0513d4, State: Initialized, Role: FOLLOWER
I20260929 16:33:32.300315 5491 consensus_queue.cc:260] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.301133 5206 heartbeater.cc:503] Master 127.23.44.62:39531 was elected leader, sending a full tablet report...
I20260929 16:33:32.301016 5492 raft_consensus.cc:348] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.301237 5492 raft_consensus.cc:374] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:32.301304 5492 raft_consensus.cc:729] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a607c1976042419803ee59b3f91cec, State: Initialized, Role: FOLLOWER
I20260929 16:33:32.301312 5491 ts_tablet_manager.cc:1428] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:33:32.301748 5495 tablet_bootstrap.cc:654] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:32.301728 5492 consensus_queue.cc:260] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.302358 5340 heartbeater.cc:503] Master 127.23.44.62:39531 was elected leader, sending a full tablet report...
I20260929 16:33:32.302588 5492 ts_tablet_manager.cc:1428] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 16:33:32.303323 5207 tablet.cc:2369] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:33:32.303634 5495 log.cc:824] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:32.305382 5497 raft_consensus.cc:482] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:32.305578 5497 raft_consensus.cc:504] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.305639 5495 tablet_bootstrap.cc:492] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665: No bootstrap required, opened a new log
I20260929 16:33:32.305825 5495 ts_tablet_manager.cc:1397] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:33:32.306600 5497 leader_election.cc:288] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141), 9647f9b96e8f4777853e35001d0513d4 (127.23.44.1:36121)
I20260929 16:33:32.309105 5495 raft_consensus.cc:348] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.309360 5495 raft_consensus.cc:374] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:32.309441 5495 raft_consensus.cc:729] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe05f378ccb40309263fec338aa1665, State: Initialized, Role: FOLLOWER
I20260929 16:33:32.309834 5495 consensus_queue.cc:260] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.310398 5474 heartbeater.cc:503] Master 127.23.44.62:39531 was elected leader, sending a full tablet report...
I20260929 16:33:32.310698 5495 ts_tablet_manager.cc:1428] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:33:32.311946 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.306757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:44141 (local address 127.23.44.2:42867)
0929 16:33:32.306876 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.306891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:32.306966 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:32.307954 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:32.307959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:32.307976 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:32.308106 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.308115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.309106 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.309112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.310565 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.310576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.311516 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.311519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.311625 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.311659 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:33:32.311741 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:33:32.311823 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":157}
I20260929 16:33:32.311981 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.307518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:42867 (local address 127.23.44.3:44141)
0929 16:33:32.307633 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:32.307638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:32.307651 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:33:32.307697 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:32.307700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:32.307742 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:32.307824 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:32.308227 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.308965 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.310706 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.311308 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.311406 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.311454 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:32.311777 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 16:33:32.311872 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 16:33:32.312624 5428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ec1a22878a45b9a44e34d93b38570b" candidate_uuid: "a2a607c1976042419803ee59b3f91cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fe05f378ccb40309263fec338aa1665" is_pre_election: true
I20260929 16:33:32.312757 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.307667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:33645 (local address 127.23.44.1:36121)
0929 16:33:32.307801 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:32.307805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:32.307841 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:33:32.307885 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:32.307888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:32.307950 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:32.308025 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:32.308288 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.308886 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.310706 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.311308 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.311396 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.311454 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:32.312563 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0929 16:33:32.312646 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:33:32.312872 5428 raft_consensus.cc:2454] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a607c1976042419803ee59b3f91cec in term 0.
I20260929 16:33:32.313098 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.306998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:36121 (local address 127.23.44.2:33645)
0929 16:33:32.307263 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.307276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:32.307363 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:32.308030 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:32.308033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:32.308048 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:32.308162 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.308170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.309057 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.309063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.310565 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.310576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.312298 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.312301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.312414 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.312438 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:33:32.312882 (+ 444us) client_negotiation.cc:240] Negotiation successful
0929 16:33:32.312961 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":110,"thread_start_us":111,"threads_started":1}
I20260929 16:33:32.313550 5229 leader_election.cc:302] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fe05f378ccb40309263fec338aa1665, a2a607c1976042419803ee59b3f91cec; no voters:
I20260929 16:33:32.313745 5160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ec1a22878a45b9a44e34d93b38570b" candidate_uuid: "a2a607c1976042419803ee59b3f91cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9647f9b96e8f4777853e35001d0513d4" is_pre_election: true
I20260929 16:33:32.313797 5497 raft_consensus.cc:2781] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:33:32.313879 5497 raft_consensus.cc:482] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:33:32.313925 5497 raft_consensus.cc:3037] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:32.314020 5160 raft_consensus.cc:2454] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a607c1976042419803ee59b3f91cec in term 0.
I20260929 16:33:32.314890 5497 raft_consensus.cc:504] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.315150 5497 leader_election.cc:288] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 1 election: Requested vote from peers 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141), 9647f9b96e8f4777853e35001d0513d4 (127.23.44.1:36121)
I20260929 16:33:32.315459 5428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ec1a22878a45b9a44e34d93b38570b" candidate_uuid: "a2a607c1976042419803ee59b3f91cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fe05f378ccb40309263fec338aa1665"
I20260929 16:33:32.315579 5428 raft_consensus.cc:3037] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:32.315596 5160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ec1a22878a45b9a44e34d93b38570b" candidate_uuid: "a2a607c1976042419803ee59b3f91cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9647f9b96e8f4777853e35001d0513d4"
I20260929 16:33:32.315712 5160 raft_consensus.cc:3037] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:32.316561 5428 raft_consensus.cc:2454] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a607c1976042419803ee59b3f91cec in term 1.
I20260929 16:33:32.316643 5160 raft_consensus.cc:2454] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a607c1976042419803ee59b3f91cec in term 1.
I20260929 16:33:32.316922 5229 leader_election.cc:302] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fe05f378ccb40309263fec338aa1665, a2a607c1976042419803ee59b3f91cec; no voters:
I20260929 16:33:32.317198 5497 raft_consensus.cc:2781] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:32.317441 5497 raft_consensus.cc:686] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 1 LEADER]: Becoming Leader. State: Replica: a2a607c1976042419803ee59b3f91cec, State: Running, Role: LEADER
I20260929 16:33:32.317723 5497 consensus_queue.cc:237] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.320250 5014 catalog_manager.cc:5495] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a607c1976042419803ee59b3f91cec (127.23.44.2). New cstate: current_term: 1 leader_uuid: "a2a607c1976042419803ee59b3f91cec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } health_report { overall_health: HEALTHY } } }
I20260929 16:33:32.342417 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.338983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.23.44.2:39867)
0929 16:33:32.339134 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:32.339140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:32.339155 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:33:32.339224 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:32.339226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:32.339270 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:32.339379 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:33:32.339626 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.340156 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.340993 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.341220 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.341368 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.341679 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:33:32.342040 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:33:32.342169 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:32.342228 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 16:33:32.342279 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:33:32.348158 5428 raft_consensus.cc:1261] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665 [term 1 FOLLOWER]: Refusing update from remote peer a2a607c1976042419803ee59b3f91cec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:32.348182 5160 raft_consensus.cc:1261] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Refusing update from remote peer a2a607c1976042419803ee59b3f91cec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:32.348891 5497 consensus_queue.cc:1035] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:32.349138 5501 consensus_queue.cc:1035] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:32.356562 5505 mvcc.cc:205] Tried to move back new op lower bound from 7334705612170534912 to 7334705612055613440. Current Snapshot: MvccSnapshot[applied={T|T < 7334705612170534912}]
I20260929 16:33:32.364472 5010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:32.364640 5010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:32.366997 5010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Sending DeleteTablet for 3 replicas of tablet 99ec1a22878a45b9a44e34d93b38570b
I20260929 16:33:32.367803 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 5077
I20260929 16:33:32.367767 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ec1a22878a45b9a44e34d93b38570b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:32 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:39638
I20260929 16:33:32.367770 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ec1a22878a45b9a44e34d93b38570b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:32 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:33752
I20260929 16:33:32.367918 5140 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ec1a22878a45b9a44e34d93b38570b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:32 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:43950
I20260929 16:33:32.369004 5514 tablet_replica.cc:330] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665: stopping tablet replica
I20260929 16:33:32.369313 5514 raft_consensus.cc:2229] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:32.369620 5514 raft_consensus.cc:2258] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:32.370775 5514 ts_tablet_manager.cc:1905] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:32.372905 5514 ts_tablet_manager.cc:1918] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:32.373009 5514 log.cc:1196] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/99ec1a22878a45b9a44e34d93b38570b
I20260929 16:33:32.373383 5514 ts_tablet_manager.cc:1939] T 99ec1a22878a45b9a44e34d93b38570b P 8fe05f378ccb40309263fec338aa1665: Deleting consensus metadata
I20260929 16:33:32.374310 5515 tablet_replica.cc:330] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: stopping tablet replica
I20260929 16:33:32.374536 5001 catalog_manager.cc:4841] TS 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141): tablet 99ec1a22878a45b9a44e34d93b38570b (table test-table [id=050a314099ce4378b3257c49e06a9131]) successfully deleted
I20260929 16:33:32.374651 5515 raft_consensus.cc:2229] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:33:32.375043 5515 raft_consensus.cc:2258] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 16:33:32.375166 5001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:36121 (error 108)
I20260929 16:33:32.376160 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.375868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:36121 (local address 127.0.0.1:43956)
0929 16:33:32.376001 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.376082 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 16:33:32.376327 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 5211
W20260929 16:33:32.376327 5001 catalog_manager.cc:4584] TS 9647f9b96e8f4777853e35001d0513d4 (127.23.44.1:36121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99ec1a22878a45b9a44e34d93b38570b: Network error: Client connection negotiation failed: client connection to 127.23.44.1:36121: connect: Connection refused (error 111)
I20260929 16:33:32.382541 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 5345
I20260929 16:33:32.382776 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.382517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:39867 (local address 127.0.0.1:39646)
0929 16:33:32.382638 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.382694 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.2:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 16:33:32.388026 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 4983
I20260929 16:33:32.394196 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39531
--webserver_interface=127.23.44.62
--webserver_port=38287
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:39531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:32.500288 5517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:32.500571 5517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:32.500641 5517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:32.504634 5517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:32.504722 5517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:32.504763 5517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:32.504822 5517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:32.504861 5517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:32.509999 5517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:39531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=38287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:32.510954 5517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:32.511904 5517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:32.524381 5523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:32.524444 5517 server_base.cc:1034] running on GCE node
W20260929 16:33:32.524402 5526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:32.524381 5524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:32.525118 5517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:32.526280 5517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:32.527441 5517 hybrid_clock.cc:648] HybridClock initialized: now 1790699612527424 us; error 29 us; skew 500 ppm
Sep 29 16:33:32 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:32.530053 5517 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:33:32.530815 5517 webserver.cc:460] Webserver started at http://127.23.44.62:38287/ using document root <none> and password file <none>
I20260929 16:33:32.531355 5517 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:32.531424 5517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:32.534952 5517 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:33:32.536937 5533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:32.538159 5517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:33:32.538254 5517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "2fda6a5dcfd740abb37c415a2d112ccd"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "1b3f841a91e9709bac7c589c4015b1b3"
server_key_iv: "ef6efa77e1a92c1c7eccd0883be5e6a4"
server_key_version: "encryptionkey@0"
I20260929 16:33:32.538771 5517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:32.549115 5517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:32.552421 5517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:32.552609 5517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:32.559865 5517 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:39531
I20260929 16:33:32.560143 5585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:39531 every 8 connection(s)
I20260929 16:33:32.560974 5517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:32.565057 5586 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:33:32.566665 5586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Bootstrap starting.
I20260929 16:33:32.569598 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 5517
I20260929 16:33:32.570003 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:36121
--local_ip_for_outbound_sockets=127.23.44.1
--tserver_master_addrs=127.23.44.62:39531
--webserver_port=46373
--webserver_interface=127.23.44.1
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:32.582944 5586 log.cc:824] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:32.591289 5586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:33:32.591636 5586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Bootstrap complete.
I20260929 16:33:32.595666 5586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } }
I20260929 16:33:32.596038 5586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fda6a5dcfd740abb37c415a2d112ccd, State: Initialized, Role: FOLLOWER
I20260929 16:33:32.596499 5586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } }
I20260929 16:33:32.596640 5586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:32.596720 5586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:32.596841 5586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:33:32.598699 5586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } }
I20260929 16:33:32.599020 5586 leader_election.cc:302] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fda6a5dcfd740abb37c415a2d112ccd; no voters:
I20260929 16:33:32.599336 5586 leader_election.cc:288] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:33:32.599411 5590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:33:32.599625 5590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 2 LEADER]: Becoming Leader. State: Replica: 2fda6a5dcfd740abb37c415a2d112ccd, State: Running, Role: LEADER
I20260929 16:33:32.599913 5590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } }
I20260929 16:33:32.600544 5586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:32.600785 5590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } } }
I20260929 16:33:32.600901 5590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:32.601233 5594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:32.601347 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fda6a5dcfd740abb37c415a2d112ccd. Latest consensus state: current_term: 2 leader_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } } }
I20260929 16:33:32.601509 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:32.605847 5594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:32.606492 5594 catalog_manager.cc:1259] Loaded cluster ID: ca6e82918fe24d70b74abe213682c0ab
I20260929 16:33:32.606566 5594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:32.607602 5594 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 16:33:32.610473 5594 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 16:33:32.614429 5594 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:32.615010 5594 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Loaded TSK: 0
I20260929 16:33:32.615931 5594 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:33:32.690536 5588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:32.690797 5588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:32.690876 5588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:32.694764 5588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:32.694846 5588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:32.694929 5588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:33:32.700003 5588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:36121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=46373
--tserver_master_addrs=127.23.44.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:32.701023 5588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:32.702174 5588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:32.712309 5614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:32.712321 5616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:32.712553 5613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:32.713255 5588 server_base.cc:1034] running on GCE node
I20260929 16:33:32.713644 5588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:32.714244 5588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:32.715469 5588 hybrid_clock.cc:648] HybridClock initialized: now 1790699612715446 us; error 39 us; skew 500 ppm
Sep 29 16:33:32 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:32.718462 5588 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:33:32.719355 5588 webserver.cc:460] Webserver started at http://127.23.44.1:46373/ using document root <none> and password file <none>
I20260929 16:33:32.719899 5588 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:32.720002 5588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:32.723963 5588 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260929 16:33:32.726347 5623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:32.727571 5588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:33:32.727711 5588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "9647f9b96e8f4777853e35001d0513d4"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "dee3a703f00e1f5e0e8a91f2b55be63e"
server_key_iv: "0b441c3f67af8c9b0528860d4b2cc0d0"
server_key_version: "encryptionkey@0"
I20260929 16:33:32.728246 5588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:32.749068 5588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:32.752225 5588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:32.752434 5588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:32.753088 5588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:32.754354 5630 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:33:32.759227 5588 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:33:32.759325 5588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 16:33:32.759419 5588 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:33:32.761108 5588 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:33:32.761180 5588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 16:33:32.761447 5630 tablet_bootstrap.cc:492] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Bootstrap starting.
I20260929 16:33:32.772257 5588 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:36121
I20260929 16:33:32.772349 5737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:36121 every 8 connection(s)
I20260929 16:33:32.773244 5588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:32.776337 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 5588
I20260929 16:33:32.776932 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:39867
--local_ip_for_outbound_sockets=127.23.44.2
--tserver_master_addrs=127.23.44.62:39531
--webserver_port=42307
--webserver_interface=127.23.44.2
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:32.780406 5630 log.cc:824] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:33:32 dist-test-slave-mlmz krb5kdc[4970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:32.787523 5630 tablet_bootstrap.cc:492] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:33:32.787938 5630 tablet_bootstrap.cc:492] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Bootstrap complete.
I20260929 16:33:32.789013 5630 ts_tablet_manager.cc:1397] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Time spent bootstrapping tablet: real 0.028s user 0.020s sys 0.004s
I20260929 16:33:32.791630 5630 raft_consensus.cc:348] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.792111 5630 raft_consensus.cc:729] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9647f9b96e8f4777853e35001d0513d4, State: Initialized, Role: FOLLOWER
I20260929 16:33:32.792629 5630 consensus_queue.cc:260] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.792806 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.775375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:43027 (local address 127.23.44.62:39531)
0929 16:33:32.775752 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:32.775771 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:32.777251 (+ 1480us) server_negotiation.cc:407] Connection header received
0929 16:33:32.777533 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:32.777544 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:32.777832 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:32.778161 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:32.779321 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.780344 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.782350 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.783030 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.783204 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.786475 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:32.786485 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:32.786507 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:32.786548 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:32.789154 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:32.789778 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:32.789789 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:32.789796 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:32.789968 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:32.790379 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:32.790384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:32.790388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:32.790888 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:32.791029 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:32.791627 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 16:33:32.791929 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":114,"threads_started":1}
I20260929 16:33:32.793175 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.775253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:43027)
0929 16:33:32.775902 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.775955 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:32.776855 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:32.778211 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:32.778219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:32.778656 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:32.779117 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.779130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.780509 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.780516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.782177 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.782192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.783198 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.783206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.783461 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.784214 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:32.784244 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:32.786259 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:32.789302 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:32.789308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:32.789320 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:32.789635 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:32.790090 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:32.790093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:32.790095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:32.790251 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:32.791017 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:32.791024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:32.791285 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 16:33:32.791465 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 16:33:32.791770 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":154,"threads_started":1}
I20260929 16:33:32.793603 5630 ts_tablet_manager.cc:1428] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260929 16:33:32.795742 5738 heartbeater.cc:347] Connected to a master server at 127.23.44.62:39531
I20260929 16:33:32.795961 5738 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:32.796392 5738 heartbeater.cc:511] Master 127.23.44.62:39531 requested a full tablet report, sending...
I20260929 16:33:32.798884 5550 ts_manager.cc:194] Registered new tserver with Master: 9647f9b96e8f4777853e35001d0513d4 (127.23.44.1:36121)
I20260929 16:33:32.802995 5550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:43027
I20260929 16:33:32.805554 5738 heartbeater.cc:503] Master 127.23.44.62:39531 was elected leader, sending a full tablet report...
Sep 29 16:33:32 dist-test-slave-mlmz krb5kdc[4970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:33:32.813112 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.802326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.23.44.1:36121)
0929 16:33:32.802894 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:32.802902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:32.803219 (+ 317us) server_negotiation.cc:407] Connection header received
0929 16:33:32.803424 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:32.803429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:32.803581 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:32.803830 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:32.804431 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.805729 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.806701 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.807154 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.807260 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.809663 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:32.809671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:32.809680 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:32.809699 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:32.811205 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:32.811732 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:32.811736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:32.811737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:32.811808 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:32.812131 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:32.812134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:32.812135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:32.812402 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:32.812510 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:32.812880 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 16:33:32.812999 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":89,"threads_started":1}
I20260929 16:33:32.813186 5745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.802234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:36121 (local address 127.0.0.1:43966)
0929 16:33:32.803078 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.803144 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:32.803285 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:32.803717 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:32.803724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:32.804170 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:32.804290 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.804298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.805869 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.805873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.806578 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.806587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.807282 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.807286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.807411 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.807925 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:32.807941 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:32.809533 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:32.811316 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:32.811326 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:32.811331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:32.811626 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:32.811897 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:32.811900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:32.811902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:32.811999 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:32.812521 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:32.812527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:32.812651 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 16:33:32.812818 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 16:33:32.813003 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":109,"threads_started":1}
I20260929 16:33:32.814334 5672 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ec1a22878a45b9a44e34d93b38570b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:32 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:43966
I20260929 16:33:32.815281 5747 tablet_replica.cc:330] stopping tablet replica
I20260929 16:33:32.815553 5747 raft_consensus.cc:2229] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:32.815775 5747 raft_consensus.cc:2258] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:32.816641 5747 ts_tablet_manager.cc:1905] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:32.819931 5747 ts_tablet_manager.cc:1918] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:32.820037 5747 log.cc:1196] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal/wals/99ec1a22878a45b9a44e34d93b38570b
I20260929 16:33:32.820297 5747 ts_tablet_manager.cc:1939] T 99ec1a22878a45b9a44e34d93b38570b P 9647f9b96e8f4777853e35001d0513d4: Deleting consensus metadata
I20260929 16:33:32.821823 5536 catalog_manager.cc:4841] TS 9647f9b96e8f4777853e35001d0513d4 (127.23.44.1:36121): tablet 99ec1a22878a45b9a44e34d93b38570b (table test-table [id=050a314099ce4378b3257c49e06a9131]) successfully deleted
W20260929 16:33:32.905663 5742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:32.905916 5742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:32.905963 5742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:32.910538 5742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:32.910596 5742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:32.910653 5742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:33:32.915731 5742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:39867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=42307
--tserver_master_addrs=127.23.44.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:32.916642 5742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:32.917619 5742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:32.926995 5755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:32.927047 5753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:32.927146 5742 server_base.cc:1034] running on GCE node
W20260929 16:33:32.927071 5752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:32.927763 5742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:32.928371 5742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:32.929565 5742 hybrid_clock.cc:648] HybridClock initialized: now 1790699612929553 us; error 25 us; skew 500 ppm
Sep 29 16:33:32 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:32.932207 5742 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:33:32.933053 5742 webserver.cc:460] Webserver started at http://127.23.44.2:42307/ using document root <none> and password file <none>
I20260929 16:33:32.933640 5742 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:32.933768 5742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:32.937381 5742 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 16:33:32.939347 5762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:32.940435 5742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:33:32.940568 5742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "a2a607c1976042419803ee59b3f91cec"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "c82c547459ed876f19ce3fe90c78a4af"
server_key_iv: "050be821fd4c6d02ae106f06d3f4295a"
server_key_version: "encryptionkey@0"
I20260929 16:33:32.941084 5742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:32.959712 5742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:32.962890 5742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:32.963101 5742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:32.963685 5742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:32.964797 5769 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:33:32.968729 5742 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:33:32.968817 5742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 16:33:32.968906 5742 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:33:32.970615 5742 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:33:32.970688 5742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 16:33:32.970892 5769 tablet_bootstrap.cc:492] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Bootstrap starting.
I20260929 16:33:32.977027 5769 log.cc:824] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:32.980508 5742 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:39867
I20260929 16:33:32.980542 5877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:39867 every 8 connection(s)
I20260929 16:33:32.981376 5742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:32.984917 5769 tablet_bootstrap.cc:492] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:33:32.985230 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 5742
I20260929 16:33:32.985399 5769 tablet_bootstrap.cc:492] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Bootstrap complete.
I20260929 16:33:32.985704 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:44141
--local_ip_for_outbound_sockets=127.23.44.3
--tserver_master_addrs=127.23.44.62:39531
--webserver_port=43041
--webserver_interface=127.23.44.3
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:32.985730 5769 ts_tablet_manager.cc:1397] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20260929 16:33:32.988663 5769 raft_consensus.cc:348] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.989032 5769 raft_consensus.cc:729] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a607c1976042419803ee59b3f91cec, State: Initialized, Role: FOLLOWER
I20260929 16:33:32.989513 5769 consensus_queue.cc:260] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:32.990509 5769 ts_tablet_manager.cc:1428] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
Sep 29 16:33:32 dist-test-slave-mlmz krb5kdc[4970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:32.995273 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.983087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:56183 (local address 127.23.44.62:39531)
0929 16:33:32.983243 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:32.983248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:32.983794 (+ 546us) server_negotiation.cc:407] Connection header received
0929 16:33:32.984863 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:32.984865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:32.984915 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:32.985055 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:32.986495 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.987274 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.988354 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.988586 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.988695 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.991693 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:32.991697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:32.991699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:32.991733 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:32.993234 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:32.993794 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:32.993799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:32.993800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:32.993890 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:32.994231 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:32.994236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:32.994238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:32.994417 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:32.994521 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:32.994984 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 16:33:32.995107 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:33:32.996013 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.983177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:56183)
0929 16:33:32.983629 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.983679 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:32.984636 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:32.985274 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:32.985286 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:32.985731 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:32.986326 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.986343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.987428 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.987431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.988215 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:32.988224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:32.988694 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:32.988697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:32.988883 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:32.990037 (+ 1154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:32.990056 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:32.991535 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:32.993420 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:32.993425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:32.993438 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:32.993690 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:32.993989 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:32.993992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:32.993994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:32.994126 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:32.994530 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:32.994539 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:32.994756 (+ 217us) client_negotiation.cc:769] Sending connection context
0929 16:33:32.994930 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 16:33:32.995215 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260929 16:33:32.997102 5878 heartbeater.cc:347] Connected to a master server at 127.23.44.62:39531
I20260929 16:33:32.997362 5878 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:32.997794 5878 heartbeater.cc:511] Master 127.23.44.62:39531 requested a full tablet report, sending...
I20260929 16:33:32.999300 5550 ts_manager.cc:194] Registered new tserver with Master: a2a607c1976042419803ee59b3f91cec (127.23.44.2:39867)
I20260929 16:33:33.000597 5550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:56183
I20260929 16:33:33.002887 5878 heartbeater.cc:503] Master 127.23.44.62:39531 was elected leader, sending a full tablet report...
Sep 29 16:33:33 dist-test-slave-mlmz krb5kdc[4970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699612, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.2@KRBTEST.COM
I20260929 16:33:33.009438 5745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.000202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:39867 (local address 127.0.0.1:39658)
0929 16:33:33.000355 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:33.000369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:33.000435 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:33.001062 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:33.001065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:33.001284 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:33.001382 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:33.001387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.002501 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.002506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:33.003154 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:33.003163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.003970 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.003974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:33.004070 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.004589 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:33.004603 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:33.005948 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:33.007805 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:33.007808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:33.007810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:33.008023 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:33.008342 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:33.008345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:33.008347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:33.008439 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:33.008891 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:33.008894 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:33.008984 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 16:33:33.009111 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 16:33:33.009253 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 16:33:33.009511 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.000300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.23.44.2:39867)
0929 16:33:33.000610 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:33.000620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:33.000640 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:33:33.000743 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:33.000748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:33.000933 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:33.001059 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:33.001494 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.002386 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.003366 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.003853 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.003973 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.006075 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:33.006083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:33.006094 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:33.006124 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:33.007659 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:33.008136 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:33.008142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:33.008144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:33.008238 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:33.008530 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:33.008534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:33.008536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:33.008785 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:33.008920 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:33.009191 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 16:33:33.009358 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":83,"threads_started":1}
I20260929 16:33:33.010360 5811 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ec1a22878a45b9a44e34d93b38570b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:32 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:39658
I20260929 16:33:33.011204 5884 tablet_replica.cc:330] stopping tablet replica
I20260929 16:33:33.011488 5884 raft_consensus.cc:2229] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:33.011725 5884 raft_consensus.cc:2258] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:33.012689 5884 ts_tablet_manager.cc:1905] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:33.016911 5884 ts_tablet_manager.cc:1918] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:33.017005 5884 log.cc:1196] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/99ec1a22878a45b9a44e34d93b38570b
I20260929 16:33:33.017287 5884 ts_tablet_manager.cc:1939] T 99ec1a22878a45b9a44e34d93b38570b P a2a607c1976042419803ee59b3f91cec: Deleting consensus metadata
I20260929 16:33:33.018891 5537 catalog_manager.cc:4841] TS a2a607c1976042419803ee59b3f91cec (127.23.44.2:39867): tablet 99ec1a22878a45b9a44e34d93b38570b (table test-table [id=050a314099ce4378b3257c49e06a9131]) successfully deleted
W20260929 16:33:33.101049 5881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:33.101409 5881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:33.101487 5881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:33.105774 5881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:33.105851 5881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:33.105966 5881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:33:33.111063 5881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:44141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=43041
--tserver_master_addrs=127.23.44.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:33.111997 5881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:33.113003 5881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:33.122126 5889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:33.122126 5892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:33.122156 5890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:33.122318 5881 server_base.cc:1034] running on GCE node
I20260929 16:33:33.122781 5881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:33.123381 5881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:33.124588 5881 hybrid_clock.cc:648] HybridClock initialized: now 1790699613124574 us; error 30 us; skew 500 ppm
Sep 29 16:33:33 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699613, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:33.127658 5881 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:33:33.128459 5881 webserver.cc:460] Webserver started at http://127.23.44.3:43041/ using document root <none> and password file <none>
I20260929 16:33:33.128996 5881 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:33.129096 5881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:33.133025 5881 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:33:33.135072 5899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:33.136435 5881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:33:33.136581 5881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "8fe05f378ccb40309263fec338aa1665"
format_stamp: "Formatted at 2026-09-29 16:33:32 on dist-test-slave-mlmz"
server_key: "aecdedbdd115f3738d76575ef5adae74"
server_key_iv: "9c27520be66f8c6710f1d0244bba6f4d"
server_key_version: "encryptionkey@0"
I20260929 16:33:33.137156 5881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:33.167153 5881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:33.169981 5881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:33.170176 5881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:33.170718 5881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:33.171485 5881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:33.171558 5881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:33.171630 5881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:33.171684 5881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:33.180195 5881 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:44141
I20260929 16:33:33.180222 6012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:44141 every 8 connection(s)
I20260929 16:33:33.181082 5881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:33:33.183409 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 5881
Sep 29 16:33:33 dist-test-slave-mlmz krb5kdc[4970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699613, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:33.191849 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.184071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.23.44.62:39531)
0929 16:33:33.184339 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:33.184345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:33.184506 (+ 161us) server_negotiation.cc:407] Connection header received
0929 16:33:33.184606 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:33.184610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:33.184672 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:33.184785 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:33.185591 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.186142 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.186995 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.187231 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.187339 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.188755 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:33.188760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:33.188762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:33.188786 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:33.190243 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:33.190656 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:33.190660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:33.190662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:33.190742 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:33.190995 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:33.191000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:33.191001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:33.191165 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:33.191367 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:33.191547 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 16:33:33.191667 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
I20260929 16:33:33.194860 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.182775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:44901 (local address 127.23.44.62:39531)
0929 16:33:33.182911 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:33.182915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:33.183820 (+ 905us) server_negotiation.cc:407] Connection header received
0929 16:33:33.185533 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:33.185536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:33.185583 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:33.185685 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:33.186741 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.187240 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.188508 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.188712 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.188828 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.191390 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:33.191394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:33.191398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:33.191427 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:33.192977 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:33.193521 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:33.193524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:33.193526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:33.193613 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:33.193991 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:33.193994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:33.193995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:33.194151 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:33.194245 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:33.194669 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 16:33:33.194755 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:33:33.195585 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.183145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:44901)
0929 16:33:33.183638 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:33.183690 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:33.185333 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:33.185857 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:33.185863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:33.186179 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:33.186605 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:33.186617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.187367 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.187371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:33.188368 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:33.188377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.188853 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.188857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:33.189045 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.189610 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:33.189630 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:33.191228 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:33.193085 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:33.193089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:33.193099 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:33.193422 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:33.193716 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:33.193719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:33.193721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:33.193857 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:33.194239 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:33.194244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:33.194460 (+ 216us) client_negotiation.cc:769] Sending connection context
0929 16:33:33.194593 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 16:33:33.194813 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":98,"threads_started":1}
I20260929 16:33:33.196730 6013 heartbeater.cc:347] Connected to a master server at 127.23.44.62:39531
I20260929 16:33:33.197065 6013 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:33.197569 6013 heartbeater.cc:511] Master 127.23.44.62:39531 requested a full tablet report, sending...
I20260929 16:33:33.198522 5550 ts_manager.cc:194] Registered new tserver with Master: 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141)
I20260929 16:33:33.199075 5550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:44901
I20260929 16:33:33.205233 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:33:33.209055 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.205888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.23.44.62:39531)
0929 16:33:33.206002 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:33.206005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:33.206193 (+ 188us) server_negotiation.cc:407] Connection header received
0929 16:33:33.206309 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:33.206311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:33.206390 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:33.206471 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:33:33.206676 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.207135 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.207938 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.208141 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.208224 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.208464 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:33:33.208785 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:33:33.208867 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:33.208932 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 16:33:33.208970 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:33:33.209693 5550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:33:33.212783 5550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:33:33.220171 5672 tablet_service.cc:1467] Processing CreateTablet for tablet 081cfe386f954eacbcccac931abe7c78 (DEFAULT_TABLE table=test-table [id=871aca20b7e24d92808d1c9431fe3abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:33.220351 5811 tablet_service.cc:1467] Processing CreateTablet for tablet 081cfe386f954eacbcccac931abe7c78 (DEFAULT_TABLE table=test-table [id=871aca20b7e24d92808d1c9431fe3abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:33.220764 5672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081cfe386f954eacbcccac931abe7c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:33.220917 5811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081cfe386f954eacbcccac931abe7c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:33.223340 5769 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Bootstrap starting.
I20260929 16:33:33.223991 5630 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: Bootstrap starting.
I20260929 16:33:33.224382 5769 tablet_bootstrap.cc:654] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:33.224859 5630 tablet_bootstrap.cc:654] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:33.225317 5745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.219312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:44141 (local address 127.0.0.1:33758)
0929 16:33:33.219532 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:33.219549 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:33.219629 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:33.220334 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:33.220339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:33.220357 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:33.220441 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:33.220447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.222055 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.222059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:33.223362 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:33.223377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.224783 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.224786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:33.224892 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.224948 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 16:33:33.225035 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:33:33.225125 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260929 16:33:33.225425 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.219524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.23.44.3:44141)
0929 16:33:33.219844 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:33.219862 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:33.219881 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:33:33.219960 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:33.219966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:33.220165 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:33.220342 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:33.220541 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.221914 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.223538 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.224638 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.224781 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.224953 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:33.225073 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 16:33:33.225217 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":148,"threads_started":1}
I20260929 16:33:33.226372 5630 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: No bootstrap required, opened a new log
I20260929 16:33:33.226462 5630 ts_tablet_manager.cc:1397] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260929 16:33:33.226874 5630 raft_consensus.cc:348] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:33.226997 5630 raft_consensus.cc:374] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:33.227041 5630 raft_consensus.cc:729] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9647f9b96e8f4777853e35001d0513d4, State: Initialized, Role: FOLLOWER
I20260929 16:33:33.227159 5630 consensus_queue.cc:260] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:33.227406 5630 ts_tablet_manager.cc:1428] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 16:33:33.228464 5947 tablet_service.cc:1467] Processing CreateTablet for tablet 081cfe386f954eacbcccac931abe7c78 (DEFAULT_TABLE table=test-table [id=871aca20b7e24d92808d1c9431fe3abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:33.229334 5947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081cfe386f954eacbcccac931abe7c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:33.229851 5769 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: No bootstrap required, opened a new log
I20260929 16:33:33.229939 5769 ts_tablet_manager.cc:1397] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 16:33:33.230379 5769 raft_consensus.cc:348] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:33.230504 5769 raft_consensus.cc:374] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:33.230540 5769 raft_consensus.cc:729] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a607c1976042419803ee59b3f91cec, State: Initialized, Role: FOLLOWER
I20260929 16:33:33.230659 5769 consensus_queue.cc:260] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:33.230960 5769 ts_tablet_manager.cc:1428] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260929 16:33:33.232286 5879 tablet.cc:2369] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:33:33.236411 6025 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Bootstrap starting.
I20260929 16:33:33.238274 6025 tablet_bootstrap.cc:654] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:33.239298 6025 log.cc:824] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:33.240986 6025 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: No bootstrap required, opened a new log
I20260929 16:33:33.241170 6025 ts_tablet_manager.cc:1397] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:33:33.244431 6025 raft_consensus.cc:348] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:33.244618 6025 raft_consensus.cc:374] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:33.244676 6025 raft_consensus.cc:729] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe05f378ccb40309263fec338aa1665, State: Initialized, Role: FOLLOWER
I20260929 16:33:33.245183 6025 consensus_queue.cc:260] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:33.245850 6013 heartbeater.cc:503] Master 127.23.44.62:39531 was elected leader, sending a full tablet report...
I20260929 16:33:33.246096 6025 ts_tablet_manager.cc:1428] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 16:33:33.274459 5739 tablet.cc:2369] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:33:33.431994 6014 tablet.cc:2369] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:33:33.553941 5882 raft_consensus.cc:482] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:33.554181 5882 raft_consensus.cc:504] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:33.555441 5882 leader_election.cc:288] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141), 9647f9b96e8f4777853e35001d0513d4 (127.23.44.1:36121)
I20260929 16:33:33.560407 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.555773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:47095 (local address 127.23.44.3:44141)
0929 16:33:33.555912 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:33.555917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:33.556089 (+ 172us) server_negotiation.cc:407] Connection header received
0929 16:33:33.556341 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:33.556344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:33.556408 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:33.556490 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:33.556767 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.557576 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.559190 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.559812 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.559917 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.559977 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:33.560193 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 16:33:33.560288 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:33:33.560400 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.555685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:44141 (local address 127.23.44.2:47095)
0929 16:33:33.555983 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:33.555995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:33.556148 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:33.556514 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:33.556518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:33.556531 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:33.556649 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:33.556655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.557681 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.557683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:33.559063 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:33.559069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.559952 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.559955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:33.560051 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.560096 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 16:33:33.560177 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:33:33.560252 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":66,"thread_start_us":90,"threads_started":1}
I20260929 16:33:33.560959 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.556220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:53449 (local address 127.23.44.1:36121)
0929 16:33:33.556478 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:33.556483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:33.556497 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:33:33.556603 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:33.556605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:33.556665 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:33.556778 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:33.556989 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.557729 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.559190 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.559822 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.559918 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.560065 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:33.560773 (+ 708us) server_negotiation.cc:299] Negotiation successful
0929 16:33:33.560861 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":60,"threads_started":1}
I20260929 16:33:33.561038 5967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081cfe386f954eacbcccac931abe7c78" candidate_uuid: "a2a607c1976042419803ee59b3f91cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fe05f378ccb40309263fec338aa1665" is_pre_election: true
I20260929 16:33:33.561187 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.556094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:36121 (local address 127.23.44.2:53449)
0929 16:33:33.556402 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:33.556414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:33.556513 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:33.556798 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:33.556800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:33.556814 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:33.556891 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:33.556895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.557826 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.557829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:33.559055 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:33.559066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.560056 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.560059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:33.560176 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.560201 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:33:33.560271 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 16:33:33.560336 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":28,"thread_start_us":57,"threads_started":1}
I20260929 16:33:33.561349 5967 raft_consensus.cc:2454] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a607c1976042419803ee59b3f91cec in term 0.
I20260929 16:33:33.561705 5692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081cfe386f954eacbcccac931abe7c78" candidate_uuid: "a2a607c1976042419803ee59b3f91cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9647f9b96e8f4777853e35001d0513d4" is_pre_election: true
I20260929 16:33:33.561866 5765 leader_election.cc:302] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fe05f378ccb40309263fec338aa1665, a2a607c1976042419803ee59b3f91cec; no voters:
I20260929 16:33:33.562001 5692 raft_consensus.cc:2454] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a607c1976042419803ee59b3f91cec in term 0.
I20260929 16:33:33.562134 5882 raft_consensus.cc:2781] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:33:33.562212 5882 raft_consensus.cc:482] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:33:33.562265 5882 raft_consensus.cc:3037] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:33.563314 5882 raft_consensus.cc:504] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:33.563614 5882 leader_election.cc:288] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 1 election: Requested vote from peers 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141), 9647f9b96e8f4777853e35001d0513d4 (127.23.44.1:36121)
I20260929 16:33:33.563885 5967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081cfe386f954eacbcccac931abe7c78" candidate_uuid: "a2a607c1976042419803ee59b3f91cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fe05f378ccb40309263fec338aa1665"
I20260929 16:33:33.563993 5692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081cfe386f954eacbcccac931abe7c78" candidate_uuid: "a2a607c1976042419803ee59b3f91cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9647f9b96e8f4777853e35001d0513d4"
I20260929 16:33:33.564006 5967 raft_consensus.cc:3037] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:33.564111 5692 raft_consensus.cc:3037] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:33.565044 5692 raft_consensus.cc:2454] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a607c1976042419803ee59b3f91cec in term 1.
I20260929 16:33:33.565227 5967 raft_consensus.cc:2454] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a607c1976042419803ee59b3f91cec in term 1.
I20260929 16:33:33.565343 5765 leader_election.cc:302] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9647f9b96e8f4777853e35001d0513d4, a2a607c1976042419803ee59b3f91cec; no voters:
I20260929 16:33:33.565497 5882 raft_consensus.cc:2781] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:33.565661 5882 raft_consensus.cc:686] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 1 LEADER]: Becoming Leader. State: Replica: a2a607c1976042419803ee59b3f91cec, State: Running, Role: LEADER
I20260929 16:33:33.565930 5882 consensus_queue.cc:237] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:33.568506 5550 catalog_manager.cc:5495] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a607c1976042419803ee59b3f91cec (127.23.44.2). New cstate: current_term: 1 leader_uuid: "a2a607c1976042419803ee59b3f91cec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } health_report { overall_health: HEALTHY } } }
I20260929 16:33:33.585320 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.581885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.23.44.2:39867)
0929 16:33:33.582174 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:33.582191 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:33.582204 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:33:33.582269 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:33.582271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:33.582319 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:33.582418 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:33:33.582618 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.583169 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.583981 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:33.584203 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:33.584292 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:33.584535 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:33:33.584908 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:33:33.584991 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:33.585089 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 16:33:33.585153 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":135,"threads_started":1}
I20260929 16:33:33.588680 5967 raft_consensus.cc:1261] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [term 1 FOLLOWER]: Refusing update from remote peer a2a607c1976042419803ee59b3f91cec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:33.588675 5692 raft_consensus.cc:1261] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Refusing update from remote peer a2a607c1976042419803ee59b3f91cec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:33.589362 5882 consensus_queue.cc:1035] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:33.589526 6031 consensus_queue.cc:1035] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:33.596679 6036 mvcc.cc:205] Tried to move back new op lower bound from 7334705617253679104 to 7334705617168359424. Current Snapshot: MvccSnapshot[applied={T|T < 7334705617253679104}]
I20260929 16:33:33.603751 5548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:33.603910 5548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:33.606135 5548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Sending DeleteTablet for 3 replicas of tablet 081cfe386f954eacbcccac931abe7c78
I20260929 16:33:33.606767 5672 tablet_service.cc:1514] Processing DeleteTablet for tablet 081cfe386f954eacbcccac931abe7c78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:33 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:43966
I20260929 16:33:33.606772 5947 tablet_service.cc:1514] Processing DeleteTablet for tablet 081cfe386f954eacbcccac931abe7c78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:33 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:33758
I20260929 16:33:33.606920 5811 tablet_service.cc:1514] Processing DeleteTablet for tablet 081cfe386f954eacbcccac931abe7c78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:33 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:39658
I20260929 16:33:33.607108 6044 tablet_replica.cc:330] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: stopping tablet replica
I20260929 16:33:33.607215 6045 tablet_replica.cc:330] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: stopping tablet replica
I20260929 16:33:33.607219 6044 raft_consensus.cc:2229] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:33.607267 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 5588
I20260929 16:33:33.607357 6044 raft_consensus.cc:2258] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:33.607815 6045 raft_consensus.cc:2229] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:33.608088 6045 raft_consensus.cc:2258] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:33.609570 6045 ts_tablet_manager.cc:1905] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:33.612226 6045 ts_tablet_manager.cc:1918] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:33.612335 6045 log.cc:1196] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/081cfe386f954eacbcccac931abe7c78
I20260929 16:33:33.612567 6046 tablet_replica.cc:330] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: stopping tablet replica
I20260929 16:33:33.612676 6045 ts_tablet_manager.cc:1939] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Deleting consensus metadata
I20260929 16:33:33.612838 6046 raft_consensus.cc:2229] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:33:33.613193 6046 raft_consensus.cc:2258] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:33.613824 5536 catalog_manager.cc:4841] TS 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141): tablet 081cfe386f954eacbcccac931abe7c78 (table test-table [id=871aca20b7e24d92808d1c9431fe3abb]) successfully deleted
I20260929 16:33:33.614084 6046 ts_tablet_manager.cc:1905] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 16:33:33.614768 5536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:36121 (error 108)
I20260929 16:33:33.615280 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 5742
I20260929 16:33:33.617542 5745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.617290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:36121 (local address 127.0.0.1:43974)
0929 16:33:33.617417 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:33.617475 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:33:33.617681 5536 catalog_manager.cc:4584] TS 9647f9b96e8f4777853e35001d0513d4 (127.23.44.1:36121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 081cfe386f954eacbcccac931abe7c78: Network error: Client connection negotiation failed: client connection to 127.23.44.1:36121: connect: Connection refused (error 111)
I20260929 16:33:33.621203 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 5881
I20260929 16:33:33.622260 5745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.621356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:39867 (local address 127.0.0.1:39668)
0929 16:33:33.622157 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:33.622201 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.2:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":732}
I20260929 16:33:33.626688 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 5517
I20260929 16:33:33.632764 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39531
--webserver_interface=127.23.44.62
--webserver_port=38287
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:39531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:33.738715 6048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:33.738965 6048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:33.739002 6048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:33.742827 6048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:33.742871 6048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:33.742890 6048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:33.742906 6048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:33.742921 6048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:33.747921 6048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:39531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:39531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=38287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:33.748800 6048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:33.749716 6048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:33:33.757643 6048 server_base.cc:1034] running on GCE node
W20260929 16:33:33.757683 6053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:33.757880 6054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:33.757880 6056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:33.758544 6048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:33.759562 6048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:33.760768 6048 hybrid_clock.cc:648] HybridClock initialized: now 1790699613760754 us; error 25 us; skew 500 ppm
Sep 29 16:33:33 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699613, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:33.763438 6048 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:33:33.764206 6048 webserver.cc:460] Webserver started at http://127.23.44.62:38287/ using document root <none> and password file <none>
I20260929 16:33:33.764719 6048 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:33.764806 6048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:33.768359 6048 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:33:33.770309 6063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:33.771376 6048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:33:33.771507 6048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "2fda6a5dcfd740abb37c415a2d112ccd"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "1b3f841a91e9709bac7c589c4015b1b3"
server_key_iv: "ef6efa77e1a92c1c7eccd0883be5e6a4"
server_key_version: "encryptionkey@0"
I20260929 16:33:33.771996 6048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:33.788312 6048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:33.791357 6048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:33.791545 6048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:33.798221 6048 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:39531
I20260929 16:33:33.798507 6115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:39531 every 8 connection(s)
I20260929 16:33:33.799288 6048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:33.803333 6116 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:33:33.804816 6116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Bootstrap starting.
I20260929 16:33:33.808535 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 6048
I20260929 16:33:33.808943 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:36121
--local_ip_for_outbound_sockets=127.23.44.1
--tserver_master_addrs=127.23.44.62:39531
--webserver_port=46373
--webserver_interface=127.23.44.1
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:33.822252 6116 log.cc:824] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:33.832964 6116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:33:33.833385 6116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd: Bootstrap complete.
I20260929 16:33:33.837181 6116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } }
I20260929 16:33:33.837620 6116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fda6a5dcfd740abb37c415a2d112ccd, State: Initialized, Role: FOLLOWER
I20260929 16:33:33.838136 6116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } }
I20260929 16:33:33.838248 6116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:33.838330 6116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:33.838444 6116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:33:33.840180 6116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } }
I20260929 16:33:33.840492 6116 leader_election.cc:302] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fda6a5dcfd740abb37c415a2d112ccd; no voters:
I20260929 16:33:33.840812 6116 leader_election.cc:288] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:33:33.840883 6120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:33:33.841101 6120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [term 3 LEADER]: Becoming Leader. State: Replica: 2fda6a5dcfd740abb37c415a2d112ccd, State: Running, Role: LEADER
I20260929 16:33:33.841400 6120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } }
I20260929 16:33:33.842027 6116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:33.842253 6120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } } }
I20260929 16:33:33.842461 6120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:33.842242 6121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fda6a5dcfd740abb37c415a2d112ccd. Latest consensus state: current_term: 3 leader_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fda6a5dcfd740abb37c415a2d112ccd" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 39531 } } }
I20260929 16:33:33.842824 6121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fda6a5dcfd740abb37c415a2d112ccd [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:33.843611 6124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:33.848367 6124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:33.848994 6124 catalog_manager.cc:1259] Loaded cluster ID: ca6e82918fe24d70b74abe213682c0ab
I20260929 16:33:33.849053 6124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 16:33:33.850071 6124 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f000835c9ad google::LogMessage::Fail() at ??:0
@ 0x7f000835f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f000835c4de google::LogMessage::Flush() at ??:0
@ 0x7f000835da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0012c9bfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0012c9cd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0012c99579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f0012cdd640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f000a5ae3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f000a5aec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f000a5b03f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f000a59f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f000bbe36db start_thread at ??:0
@ 0x7f0006e5a61f clone at ??:0
W20260929 16:33:33.928270 6118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:33.928506 6118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:33.928606 6118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:33.932535 6118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:33.932617 6118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:33.932719 6118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:33:33.937868 6118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:36121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=46373
--tserver_master_addrs=127.23.44.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:33.938802 6118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:33.939710 6118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:33.948804 6158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:33.948776 6157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:33.948882 6160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:33.949281 6118 server_base.cc:1034] running on GCE node
I20260929 16:33:33.949667 6118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:33.950273 6118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:33.951457 6118 hybrid_clock.cc:648] HybridClock initialized: now 1790699613951440 us; error 53 us; skew 500 ppm
Sep 29 16:33:33 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699613, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:33.954330 6118 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:33:33.955235 6118 webserver.cc:460] Webserver started at http://127.23.44.1:46373/ using document root <none> and password file <none>
I20260929 16:33:33.955765 6118 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:33.955935 6118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:33.959812 6118 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:33:33.961936 6167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:33.963243 6118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:33:33.963399 6118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "9647f9b96e8f4777853e35001d0513d4"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "dee3a703f00e1f5e0e8a91f2b55be63e"
server_key_iv: "0b441c3f67af8c9b0528860d4b2cc0d0"
server_key_version: "encryptionkey@0"
I20260929 16:33:33.963960 6118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:34.007414 6118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:34.010725 6118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:34.010929 6118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:34.011516 6118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:34.012606 6174 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:33:34.018074 6118 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:33:34.018162 6118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 16:33:34.018250 6118 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:33:34.019924 6118 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:33:34.020004 6118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 16:33:34.020347 6174 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: Bootstrap starting.
I20260929 16:33:34.031599 6118 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:36121
I20260929 16:33:34.031672 6281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:36121 every 8 connection(s)
I20260929 16:33:34.032585 6118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:34.035686 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 6118
I20260929 16:33:34.036196 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:39867
--local_ip_for_outbound_sockets=127.23.44.2
--tserver_master_addrs=127.23.44.62:39531
--webserver_port=42307
--webserver_interface=127.23.44.2
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:34.043947 6174 log.cc:824] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:34.050779 6174 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:33:34.051151 6174 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: Bootstrap complete.
I20260929 16:33:34.052318 6174 ts_tablet_manager.cc:1397] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.011s
I20260929 16:33:34.055121 6174 raft_consensus.cc:348] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:34.055578 6174 raft_consensus.cc:729] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9647f9b96e8f4777853e35001d0513d4, State: Initialized, Role: FOLLOWER
I20260929 16:33:34.056056 6174 consensus_queue.cc:260] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:34.057009 6174 ts_tablet_manager.cc:1428] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260929 16:33:34.135941 6284 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:33:34.034513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:60225)
0929 16:33:34.035004 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.035058 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:34.036663 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:34.135226 (+ 98563us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
W20260929 16:33:34.136511 6282 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:39531 (0 consecutive failures): Network error: Failed to ping master at 127.23.44.62:39531: Client connection negotiation failed: client connection to 127.23.44.62:39531: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:33:34.137212 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.136968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:55729)
0929 16:33:34.137091 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.137161 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:33:34.137897 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.137693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:48347)
0929 16:33:34.137802 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.137851 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 16:33:34.138083 6282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:33:34.175894 6285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:34.176139 6285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:34.176216 6285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:34.180178 6285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:34.180256 6285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:34.180339 6285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:33:34.185335 6285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:39867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=42307
--tserver_master_addrs=127.23.44.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:34.186215 6285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:34.187126 6285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:34.196002 6292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:34.196031 6295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:34.196053 6293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:34.196290 6285 server_base.cc:1034] running on GCE node
I20260929 16:33:34.196658 6285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:34.197346 6285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:34.198587 6285 hybrid_clock.cc:648] HybridClock initialized: now 1790699614198550 us; error 50 us; skew 500 ppm
Sep 29 16:33:34 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699614, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:34.206700 6285 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:33:34.207592 6285 webserver.cc:460] Webserver started at http://127.23.44.2:42307/ using document root <none> and password file <none>
I20260929 16:33:34.208135 6285 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:34.208227 6285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:34.211875 6285 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:33:34.214061 6302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:34.215337 6285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:33:34.215480 6285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "a2a607c1976042419803ee59b3f91cec"
format_stamp: "Formatted at 2026-09-29 16:33:31 on dist-test-slave-mlmz"
server_key: "c82c547459ed876f19ce3fe90c78a4af"
server_key_iv: "050be821fd4c6d02ae106f06d3f4295a"
server_key_version: "encryptionkey@0"
I20260929 16:33:34.216034 6285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:34.240490 6285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:34.244388 6285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:34.244621 6285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:34.245328 6285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:34.246495 6309 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:33:34.250564 6285 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:33:34.250629 6285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 16:33:34.250708 6285 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:33:34.252292 6285 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:33:34.252334 6285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 16:33:34.252686 6309 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Bootstrap starting.
I20260929 16:33:34.258755 6309 log.cc:824] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:34.262295 6285 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:39867
I20260929 16:33:34.262362 6417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:39867 every 8 connection(s)
I20260929 16:33:34.263103 6285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:34.266201 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.264972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:53827)
0929 16:33:34.265505 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.265600 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":157,"threads_started":1}
I20260929 16:33:34.266199 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 6285
W20260929 16:33:34.266587 6418 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:39531 (0 consecutive failures): Network error: Failed to ping master at 127.23.44.62:39531: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
I20260929 16:33:34.266654 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:44141
--local_ip_for_outbound_sockets=127.23.44.3
--tserver_master_addrs=127.23.44.62:39531
--webserver_port=43041
--webserver_interface=127.23.44.3
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:34.266768 6309 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:33:34.267154 6309 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Bootstrap complete.
I20260929 16:33:34.267498 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.266953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:49071)
0929 16:33:34.267259 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.267315 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260929 16:33:34.267561 6309 ts_tablet_manager.cc:1397] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.008s
I20260929 16:33:34.268298 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.268007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:43087)
0929 16:33:34.268178 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.268228 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260929 16:33:34.268486 6418 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:33:34.270454 6309 raft_consensus.cc:348] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:34.270851 6309 raft_consensus.cc:729] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a607c1976042419803ee59b3f91cec, State: Initialized, Role: FOLLOWER
I20260929 16:33:34.271332 6309 consensus_queue.cc:260] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:34.272322 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.272096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:40433)
0929 16:33:34.272209 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.272257 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 16:33:34.272348 6309 ts_tablet_manager.cc:1428] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:33:34.273030 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.272746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:51137)
0929 16:33:34.272890 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.272954 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:33:34.379643 6422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:34.379983 6422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:34.380270 6422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:34.387579 6422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:34.387648 6422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:34.387750 6422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:33:34.396154 6422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:44141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=43041
--tserver_master_addrs=127.23.44.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:34.397392 6422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:34.398720 6422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:33:34.408413 6422 server_base.cc:1034] running on GCE node
W20260929 16:33:34.408409 6429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:34.408581 6428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:34.413718 6431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:34.414141 6422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:34.414757 6422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:34.415921 6422 hybrid_clock.cc:648] HybridClock initialized: now 1790699614415909 us; error 24 us; skew 500 ppm
Sep 29 16:33:34 dist-test-slave-mlmz krb5kdc[4970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699614, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:34.418596 6422 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:33:34.419389 6422 webserver.cc:460] Webserver started at http://127.23.44.3:43041/ using document root <none> and password file <none>
I20260929 16:33:34.420011 6422 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:34.420099 6422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:34.423749 6422 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:33:34.425776 6438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:34.426944 6422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:33:34.427168 6422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "8fe05f378ccb40309263fec338aa1665"
format_stamp: "Formatted at 2026-09-29 16:33:32 on dist-test-slave-mlmz"
server_key: "aecdedbdd115f3738d76575ef5adae74"
server_key_iv: "9c27520be66f8c6710f1d0244bba6f4d"
server_key_version: "encryptionkey@0"
I20260929 16:33:34.427673 6422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:34.445180 6422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:34.447976 6422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:34.448170 6422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:34.448706 6422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:34.449581 6422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:34.449664 6422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:34.449728 6422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:34.449777 6422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:34.459146 6422 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:44141
I20260929 16:33:34.459183 6551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:44141 every 8 connection(s)
I20260929 16:33:34.460068 6422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:33:34.463675 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.462070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:35649)
0929 16:33:34.462516 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.462620 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":109,"threads_started":1}
W20260929 16:33:34.464093 6552 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:39531 (0 consecutive failures): Network error: Failed to ping master at 127.23.44.62:39531: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
I20260929 16:33:34.464600 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.464414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:52231)
0929 16:33:34.464508 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.464552 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 16:33:34.465014 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 6422
I20260929 16:33:34.465342 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.465089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:33037)
0929 16:33:34.465210 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.465249 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 16:33:34.465544 6552 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:33:35.139042 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.138613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:40407)
0929 16:33:35.138875 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:35.138926 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":70,"threads_started":1}
I20260929 16:33:35.274044 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.273677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:33539)
0929 16:33:35.273909 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:35.273962 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260929 16:33:35.437005 6558 raft_consensus.cc:482] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:35.437206 6558 raft_consensus.cc:504] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:35.438323 6558 leader_election.cc:288] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141), 9647f9b96e8f4777853e35001d0513d4 (127.23.44.1:36121)
Sep 29 16:33:35 dist-test-slave-mlmz krb5kdc[4970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699614, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
Sep 29 16:33:35 dist-test-slave-mlmz krb5kdc[4970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699614, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.3@KRBTEST.COM
I20260929 16:33:35.451638 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.439145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:35975 (local address 127.23.44.1:36121)
0929 16:33:35.439530 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:35.439542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:35.439562 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:33:35.439674 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:35.439678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:35.439860 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:35.440044 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:35.441173 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.442244 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.443521 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.443995 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.444124 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:35.447116 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:35.447122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:35.447137 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:35.447181 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:35.448990 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:35.449630 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:35.449637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:35.449642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:35.449796 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:35.450181 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:35.450184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:35.450186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:35.450599 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:35.450699 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:35.451249 (+ 550us) server_negotiation.cc:299] Negotiation successful
0929 16:33:35.451462 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":71,"threads_started":1}
I20260929 16:33:35.451645 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.438517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:44141 (local address 127.23.44.2:50385)
0929 16:33:35.438691 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:35.438723 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:35.439213 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:35.440515 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:35.440518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:35.440735 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:35.441001 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:35.441013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.442413 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.442417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:35.443362 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:35.443371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.444128 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.444132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:35.444321 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:35.444985 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:35.444998 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:35.447288 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:35.449529 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:35.449531 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:35.449532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:35.449796 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:35.450166 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:35.450169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:35.450171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:35.450268 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:35.450849 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:35.450851 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:35.450959 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 16:33:35.451134 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 16:33:35.451443 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":82,"spinlock_wait_cycles":4736}
I20260929 16:33:35.451694 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.438721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:50385 (local address 127.23.44.3:44141)
0929 16:33:35.439346 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:35.439361 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:35.439405 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:33:35.440045 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:35.440049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:35.440315 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:35.440576 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:35.441156 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.442281 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.443479 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.443983 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.444174 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:35.447459 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:35.447464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:35.447473 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:35.447529 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:35.449398 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:35.449920 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:35.449926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:35.449931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:35.450054 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:35.450362 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:35.450365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:35.450366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:35.450742 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:35.450863 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:35.451242 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 16:33:35.451503 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":95,"threads_started":1}
I20260929 16:33:35.451696 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.438800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:36121 (local address 127.23.44.2:35975)
0929 16:33:35.439307 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:35.439321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:35.439420 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:35.440120 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:35.440130 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:35.440582 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:35.440999 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:35.441012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.442410 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.442413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:35.443370 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:35.443379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.444343 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.444346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:35.444427 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:35.444959 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:35.444990 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:35.446909 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:35.449139 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:35.449146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:35.449159 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:35.449513 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:35.449908 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:35.449911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:35.449912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:35.450063 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:35.450716 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:35.450722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:35.450974 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 16:33:35.451135 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:33:35.451527 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":188,"spinlock_wait_cycles":7680,"thread_start_us":52,"threads_started":1}
I20260929 16:33:35.453241 6236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081cfe386f954eacbcccac931abe7c78" candidate_uuid: "a2a607c1976042419803ee59b3f91cec" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9647f9b96e8f4777853e35001d0513d4" is_pre_election: true
I20260929 16:33:35.453593 6236 raft_consensus.cc:2454] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a607c1976042419803ee59b3f91cec in term 1.
W20260929 16:33:35.454124 6305 leader_election.cc:341] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141): Not found: Tablet not found: 081cfe386f954eacbcccac931abe7c78
I20260929 16:33:35.454388 6305 leader_election.cc:302] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9647f9b96e8f4777853e35001d0513d4, a2a607c1976042419803ee59b3f91cec; no voters: 8fe05f378ccb40309263fec338aa1665
I20260929 16:33:35.454679 6558 raft_consensus.cc:2781] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 16:33:35.454756 6558 raft_consensus.cc:482] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:33:35.454808 6558 raft_consensus.cc:3037] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:33:35.456827 6558 raft_consensus.cc:504] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:35.457180 6558 leader_election.cc:288] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 2 election: Requested vote from peers 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141), 9647f9b96e8f4777853e35001d0513d4 (127.23.44.1:36121)
I20260929 16:33:35.457656 6236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081cfe386f954eacbcccac931abe7c78" candidate_uuid: "a2a607c1976042419803ee59b3f91cec" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9647f9b96e8f4777853e35001d0513d4"
I20260929 16:33:35.457790 6236 raft_consensus.cc:3037] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 1 FOLLOWER]: Advancing to term 2
W20260929 16:33:35.457830 6305 leader_election.cc:341] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141): Not found: Tablet not found: 081cfe386f954eacbcccac931abe7c78
I20260929 16:33:35.459733 6236 raft_consensus.cc:2454] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a607c1976042419803ee59b3f91cec in term 2.
I20260929 16:33:35.460047 6305 leader_election.cc:302] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9647f9b96e8f4777853e35001d0513d4, a2a607c1976042419803ee59b3f91cec; no voters: 8fe05f378ccb40309263fec338aa1665
I20260929 16:33:35.460264 6558 raft_consensus.cc:2781] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:33:35.460465 6558 raft_consensus.cc:686] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [term 2 LEADER]: Becoming Leader. State: Replica: a2a607c1976042419803ee59b3f91cec, State: Running, Role: LEADER
I20260929 16:33:35.460767 6558 consensus_queue.cc:237] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:35.461567 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.461369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:48193)
0929 16:33:35.461455 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:35.461510 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 16:33:35.466284 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.465954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:58869)
0929 16:33:35.466166 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:35.466211 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
W20260929 16:33:35.922750 6305 consensus_peers.cc:487] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec -> Peer 8fe05f378ccb40309263fec338aa1665 (127.23.44.3:44141): Couldn't send request to peer 8fe05f378ccb40309263fec338aa1665. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 081cfe386f954eacbcccac931abe7c78. This is attempt 1: this message will repeat every 5th retry.
I20260929 16:33:36.039963 6236 raft_consensus.cc:1261] T 081cfe386f954eacbcccac931abe7c78 P 9647f9b96e8f4777853e35001d0513d4 [term 2 FOLLOWER]: Refusing update from remote peer a2a607c1976042419803ee59b3f91cec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 16:33:36.040655 6558 consensus_queue.cc:1035] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 16:33:36.042616 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:36.042266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:33091)
0929 16:33:36.042465 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:36.042524 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":56,"threads_started":1}
I20260929 16:33:36.044858 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:36.044488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:57053)
0929 16:33:36.044687 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:36.044749 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260929 16:33:36.045354 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:36.045114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:56577)
0929 16:33:36.045217 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:36.045293 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 16:33:36.446784 6572 ts_tablet_manager.cc:927] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Initiating tablet copy from peer a2a607c1976042419803ee59b3f91cec (127.23.44.2:39867)
I20260929 16:33:36.447423 6572 tablet_copy_client.cc:316] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: tablet copy: Beginning tablet copy session from remote peer at address 127.23.44.2:39867
Sep 29 16:33:36 dist-test-slave-mlmz krb5kdc[4970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699614, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.2@KRBTEST.COM
I20260929 16:33:36.457813 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:36.447938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:39867 (local address 127.23.44.3:58111)
0929 16:33:36.448215 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:36.448266 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:36.448425 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:36.448927 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:36.448933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:36.449421 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:36.449551 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:36.449559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:36.450642 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:36.450648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:36.451376 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:36.451386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:36.452033 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:36.452037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:36.452179 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:36.452739 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:36.452751 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:36.454148 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:36.456293 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:36.456301 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:36.456305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:36.456527 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:36.456804 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:36.456806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:36.456808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:36.456882 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:36.457338 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:36.457343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:36.457447 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 16:33:36.457565 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:33:36.457690 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
I20260929 16:33:36.457957 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:36.448110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:58111 (local address 127.23.44.2:39867)
0929 16:33:36.448485 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:36.448501 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:36.448527 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:33:36.448586 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:36.448591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:36.448769 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:36.448926 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:36.449703 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:36.450508 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:36.451516 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:36.451904 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:36.452034 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:36.454296 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:36.454304 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:36.454314 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:36.454337 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:36.456196 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:36.456626 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:36.456630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:36.456631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:36.456713 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:36.456960 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:36.456963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:36.456964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:36.457191 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:36.457370 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:36.457645 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 16:33:36.457775 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
I20260929 16:33:36.458984 6392 tablet_copy_service.cc:140] P a2a607c1976042419803ee59b3f91cec: Received BeginTabletCopySession request for tablet 081cfe386f954eacbcccac931abe7c78 from peer 8fe05f378ccb40309263fec338aa1665 ({username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:58111)
I20260929 16:33:36.459229 6392 tablet_copy_service.cc:161] P a2a607c1976042419803ee59b3f91cec: Beginning new tablet copy session on tablet 081cfe386f954eacbcccac931abe7c78 from peer 8fe05f378ccb40309263fec338aa1665 at {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:58111: session id = 8fe05f378ccb40309263fec338aa1665-081cfe386f954eacbcccac931abe7c78
I20260929 16:33:36.460362 6392 tablet_copy_source_session.cc:205] T 081cfe386f954eacbcccac931abe7c78 P a2a607c1976042419803ee59b3f91cec: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:33:36.464001 6572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081cfe386f954eacbcccac931abe7c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:36.467103 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:36.466847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:54137)
0929 16:33:36.466976 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:36.467037 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:33:36.470525 6572 tablet_copy_client.cc:791] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: tablet copy: Starting download of 0 data blocks...
I20260929 16:33:36.470718 6572 tablet_copy_client.cc:655] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: tablet copy: Starting download of 1 WAL segments...
I20260929 16:33:36.472303 6572 tablet_copy_client.cc:531] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:33:36.473790 6572 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Bootstrap starting.
I20260929 16:33:36.478752 6572 log.cc:824] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:36.484866 6572 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:33:36.485167 6572 tablet_bootstrap.cc:492] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Bootstrap complete.
I20260929 16:33:36.485488 6572 ts_tablet_manager.cc:1397] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Time spent bootstrapping tablet: real 0.012s user 0.012s sys 0.000s
I20260929 16:33:36.487648 6572 raft_consensus.cc:348] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:36.487951 6572 raft_consensus.cc:729] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe05f378ccb40309263fec338aa1665, State: Initialized, Role: FOLLOWER
I20260929 16:33:36.488418 6572 consensus_queue.cc:260] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fe05f378ccb40309263fec338aa1665" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 44141 } } peers { permanent_uuid: "9647f9b96e8f4777853e35001d0513d4" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 36121 } } peers { permanent_uuid: "a2a607c1976042419803ee59b3f91cec" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 39867 } }
I20260929 16:33:36.489210 6572 ts_tablet_manager.cc:1428] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 16:33:36.489542 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:36.489231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:59869)
0929 16:33:36.489411 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:36.489464 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 16:33:36.489915 6392 tablet_copy_service.cc:342] P a2a607c1976042419803ee59b3f91cec: Request end of tablet copy session 8fe05f378ccb40309263fec338aa1665-081cfe386f954eacbcccac931abe7c78 received from {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:58111
I20260929 16:33:36.490028 6392 tablet_copy_service.cc:433] P a2a607c1976042419803ee59b3f91cec: ending tablet copy session 8fe05f378ccb40309263fec338aa1665-081cfe386f954eacbcccac931abe7c78 on tablet 081cfe386f954eacbcccac931abe7c78 with peer 8fe05f378ccb40309263fec338aa1665
I20260929 16:33:36.490262 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:36.490011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:60267)
0929 16:33:36.490150 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:36.490204 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:33:36.864758 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:36.864394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:41141)
0929 16:33:36.864609 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:36.864659 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260929 16:33:36.864706 6506 raft_consensus.cc:1203] T 081cfe386f954eacbcccac931abe7c78 P 8fe05f378ccb40309263fec338aa1665 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260929 16:33:36.865693 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:36.865424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:56039)
0929 16:33:36.865580 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:36.865635 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 16:33:37.046307 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:37.045944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:50787)
0929 16:33:37.046144 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:37.046194 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":58,"threads_started":1}
I20260929 16:33:37.866671 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:37.866203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:44399)
0929 16:33:37.866495 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:37.866567 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
I20260929 16:33:37.866757 6582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:37.866310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:56997)
0929 16:33:37.866600 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:37.866656 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":58,"threads_started":1}
I20260929 16:33:38.047462 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:38.047055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:43601)
0929 16:33:38.047319 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:38.047378 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
I20260929 16:33:38.868086 6585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:38.867639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:46467)
0929 16:33:38.867900 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:38.867989 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
I20260929 16:33:38.868114 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:38.867597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:60037)
0929 16:33:38.867899 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:38.867982 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260929 16:33:39.048810 6586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.048395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:56001)
0929 16:33:39.048656 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:39.048720 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260929 16:33:39.869506 6588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.869027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:42431)
0929 16:33:39.869367 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:39.869416 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":105,"threads_started":1}
I20260929 16:33:39.869525 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.869026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:35121)
0929 16:33:39.869385 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:39.869441 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":88,"threads_started":1}
I20260929 16:33:39.870237 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.870014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:58447)
0929 16:33:39.870112 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:39.870155 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 16:33:40.050045 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:40.049605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:56931)
0929 16:33:40.049903 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:40.049958 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":88,"threads_started":1}
I20260929 16:33:40.870544 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:40.870158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:42811)
0929 16:33:40.870400 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:40.870453 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260929 16:33:40.871165 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:40.870827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:45877)
0929 16:33:40.871039 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:40.871086 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260929 16:33:41.051065 6592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:41.050678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:53067)
0929 16:33:41.050927 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:41.050980 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260929 16:33:41.871613 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:41.871215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:53651)
0929 16:33:41.871452 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:41.871504 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260929 16:33:41.872180 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:41.871848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:49255)
0929 16:33:41.872055 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:41.872105 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":62,"threads_started":1}
I20260929 16:33:42.052062 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.051708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:39773)
0929 16:33:42.051927 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.051978 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
I20260929 16:33:42.872762 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.872296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:46065)
0929 16:33:42.872571 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.872646 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":67,"threads_started":1}
I20260929 16:33:42.873165 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.872798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:47851)
0929 16:33:42.873043 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.873090 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260929 16:33:43.053076 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:43.052688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:58527)
0929 16:33:43.052936 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:43.052988 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":68,"threads_started":1}
I20260929 16:33:43.873893 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:43.873502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:47723)
0929 16:33:43.873751 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:43.873801 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
I20260929 16:33:43.874322 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:43.873918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:56473)
0929 16:33:43.874195 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:43.874246 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260929 16:33:44.054255 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:44.053799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:37597)
0929 16:33:44.054106 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:44.054168 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":111,"threads_started":1}
I20260929 16:33:44.875320 6602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:44.874796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:58437)
0929 16:33:44.875123 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:44.875184 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":88,"threads_started":1}
I20260929 16:33:44.875592 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:44.875161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:58983)
0929 16:33:44.875421 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:44.875493 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260929 16:33:45.055269 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:45.054885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:35767)
0929 16:33:45.055113 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:45.055178 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260929 16:33:45.876537 6605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:45.876055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:40775)
0929 16:33:45.876353 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:45.876413 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
I20260929 16:33:45.876799 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:45.876401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:58931)
0929 16:33:45.876636 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:45.876704 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
I20260929 16:33:46.056327 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.055953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:55103)
0929 16:33:46.056182 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:46.056241 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
I20260929 16:33:46.877858 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.877332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:47697)
0929 16:33:46.877612 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:46.877679 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":70,"threads_started":1}
I20260929 16:33:46.877949 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.877509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:49473)
0929 16:33:46.877739 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:46.877828 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260929 16:33:47.057404 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.056980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:40401)
0929 16:33:47.057222 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:47.057323 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":64,"threads_started":1}
I20260929 16:33:47.879087 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.878685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:33149)
0929 16:33:47.878941 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:47.878993 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260929 16:33:47.879112 6611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.878680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:59407)
0929 16:33:47.878985 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:47.879032 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":88,"threads_started":1}
I20260929 16:33:48.058452 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:48.058068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:48243)
0929 16:33:48.058307 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:48.058362 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260929 16:33:48.880335 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:48.879915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.3:55625)
0929 16:33:48.880185 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:48.880240 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
I20260929 16:33:48.880367 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:48.879918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.2:33599)
0929 16:33:48.880229 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:48.880282 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":93,"threads_started":1}
I20260929 16:33:49.059543 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:49.059188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:39531 (local address 127.23.44.1:41543)
0929 16:33:49.059401 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:49.059449 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:39531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260929 16:33:49.469426 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 6118
I20260929 16:33:49.475009 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 6285
I20260929 16:33:49.480172 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 6422
2026-09-29T16:33:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19261 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:33:49 dist-test-slave-mlmz krb5kdc[6619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:33:49 dist-test-slave-mlmz krb5kdc[6619](info): set up 2 sockets
Sep 29 16:33:49 dist-test-slave-mlmz krb5kdc[6619](info): commencing operation
krb5kdc: starting...
W20260929 16:33:50.521879 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:33:50 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:33:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:33:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:33:50.640198 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:38275
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:38275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:50.746098 6632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:50.746366 6632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:50.746434 6632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:50.750315 6632 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:33:50.750383 6632 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:33:50.750417 6632 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:33:50.750447 6632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:50.750488 6632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:50.750525 6632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:50.750566 6632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:50.750600 6632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:50.755725 6632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:38275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:38275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:50.756687 6632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:50.757647 6632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:50.765959 6638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:50.765969 6640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:50.766006 6632 server_base.cc:1034] running on GCE node
W20260929 16:33:50.765959 6637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:50.766712 6632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:50.767710 6632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:50.768867 6632 hybrid_clock.cc:648] HybridClock initialized: now 1790699630768847 us; error 31 us; skew 500 ppm
Sep 29 16:33:50 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699630, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:50.771761 6632 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:33:50.772657 6632 webserver.cc:460] Webserver started at http://127.23.44.62:36799/ using document root <none> and password file <none>
I20260929 16:33:50.773183 6632 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:50.773243 6632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:50.773504 6632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:50.775560 6632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "96713613bfee4618ac4b7b48a57622e8"
format_stamp: "Formatted at 2026-09-29 16:33:50 on dist-test-slave-mlmz"
server_key: "3c1517f1fb409450faa8c0dbc9758fd1"
server_key_iv: "f7b641f0d1cba1dcf3e23f092432e81a"
server_key_version: "encryptionkey@0"
I20260929 16:33:50.776111 6632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "96713613bfee4618ac4b7b48a57622e8"
format_stamp: "Formatted at 2026-09-29 16:33:50 on dist-test-slave-mlmz"
server_key: "3c1517f1fb409450faa8c0dbc9758fd1"
server_key_iv: "f7b641f0d1cba1dcf3e23f092432e81a"
server_key_version: "encryptionkey@0"
I20260929 16:33:50.780587 6632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 16:33:50.783370 6647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:50.784682 6632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:33:50.784879 6632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "96713613bfee4618ac4b7b48a57622e8"
format_stamp: "Formatted at 2026-09-29 16:33:50 on dist-test-slave-mlmz"
server_key: "3c1517f1fb409450faa8c0dbc9758fd1"
server_key_iv: "f7b641f0d1cba1dcf3e23f092432e81a"
server_key_version: "encryptionkey@0"
I20260929 16:33:50.784991 6632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:50.807592 6632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:50.810629 6632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:50.810814 6632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:50.817184 6632 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:38275
I20260929 16:33:50.817493 6699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:38275 every 8 connection(s)
I20260929 16:33:50.818087 6632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:50.820693 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:50.825542 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 6632
I20260929 16:33:50.825668 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:33:50.825968 23728 external_mini_cluster.cc:1428] Setting key 163f3ddbd16abe7ad082eaf1e35fa5fb
I20260929 16:33:50.826208 6700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Bootstrap starting.
I20260929 16:33:50.828617 6700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:50.829375 6700 log.cc:824] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:50.831084 6700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: No bootstrap required, opened a new log
Sep 29 16:33:50 dist-test-slave-mlmz krb5kdc[6619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:50.834925 6700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } }
I20260929 16:33:50.835155 6700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:50.835234 6700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96713613bfee4618ac4b7b48a57622e8, State: Initialized, Role: FOLLOWER
I20260929 16:33:50.835700 6700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } }
I20260929 16:33:50.835865 6700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:50.835935 6700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:50.836035 6700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:50.836907 6700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } }
I20260929 16:33:50.837288 6700 leader_election.cc:302] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96713613bfee4618ac4b7b48a57622e8; no voters:
I20260929 16:33:50.837618 6700 leader_election.cc:288] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:33:50.837685 6705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:50.837863 6705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 1 LEADER]: Becoming Leader. State: Replica: 96713613bfee4618ac4b7b48a57622e8, State: Running, Role: LEADER
I20260929 16:33:50.838141 6705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } }
I20260929 16:33:50.838802 6700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:50.839211 6705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96713613bfee4618ac4b7b48a57622e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } } }
I20260929 16:33:50.839319 6705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:50.839519 6705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96713613bfee4618ac4b7b48a57622e8. Latest consensus state: current_term: 1 leader_uuid: "96713613bfee4618ac4b7b48a57622e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } } }
I20260929 16:33:50.839633 6705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:50.839768 6709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:50.840281 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.827436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.23.44.62:38275)
0929 16:33:50.827856 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:50.827871 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:50.827904 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:33:50.828501 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:50.828516 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:50.828872 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:50.829199 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:50.830111 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.831065 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.831896 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.832343 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.832517 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:50.835195 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:50.835201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:50.835218 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:50.835249 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:50.837052 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:50.837566 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:50.837572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:50.837577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:50.837775 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:50.838042 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:50.838045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:50.838047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:50.838485 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:50.838694 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:50.838918 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 16:33:50.839227 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":142,"threads_started":1}
I20260929 16:33:50.843638 6709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:50.848693 6709 catalog_manager.cc:1347] Generated new cluster ID: 4f8968dc40864761b79e657f53ebfd33
I20260929 16:33:50.848757 6709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:50.864516 6709 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:33:50.865382 6709 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:50.872527 6709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Generated new TSK 0
I20260929 16:33:50.873135 6709 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:33:50.904201 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38275
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:51.010726 6726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:51.010941 6726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:51.010979 6726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:51.014783 6726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:51.014824 6726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:51.014878 6726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:33:51.019949 6726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:51.020750 6726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:51.021656 6726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:51.030475 6732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:51.030488 6734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:51.030506 6731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:51.030632 6726 server_base.cc:1034] running on GCE node
I20260929 16:33:51.031147 6726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:51.031716 6726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:51.032873 6726 hybrid_clock.cc:648] HybridClock initialized: now 1790699631032856 us; error 31 us; skew 500 ppm
Sep 29 16:33:51 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699631, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:51.035547 6726 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:33:51.036311 6726 webserver.cc:460] Webserver started at http://127.23.44.1:40051/ using document root <none> and password file <none>
I20260929 16:33:51.036788 6726 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:51.036844 6726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:51.036990 6726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:51.038853 6726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "b60e9e128c264e39b4ba808462bb7f20"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "20ecd2c0a8e5bfe5d6d59e4e0199ad0f"
server_key_iv: "f400adbdb855c06d56d9b491de81ab56"
server_key_version: "encryptionkey@0"
I20260929 16:33:51.039463 6726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "b60e9e128c264e39b4ba808462bb7f20"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "20ecd2c0a8e5bfe5d6d59e4e0199ad0f"
server_key_iv: "f400adbdb855c06d56d9b491de81ab56"
server_key_version: "encryptionkey@0"
I20260929 16:33:51.042991 6726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 16:33:51.045147 6741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:51.046317 6726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:33:51.046447 6726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "b60e9e128c264e39b4ba808462bb7f20"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "20ecd2c0a8e5bfe5d6d59e4e0199ad0f"
server_key_iv: "f400adbdb855c06d56d9b491de81ab56"
server_key_version: "encryptionkey@0"
I20260929 16:33:51.046548 6726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:51.059723 6726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:51.062672 6726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:51.062849 6726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:51.063365 6726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:51.064083 6726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:51.064148 6726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:51.064215 6726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:51.064254 6726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:51.072242 6726 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:44873
I20260929 16:33:51.072273 6854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:44873 every 8 connection(s)
I20260929 16:33:51.073017 6726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:51.079493 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 6726
I20260929 16:33:51.079617 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:33:51.079917 23728 external_mini_cluster.cc:1428] Setting key 0ac6f8ea82cf95cffcffb4642bb38725
Sep 29 16:33:51 dist-test-slave-mlmz krb5kdc[6619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699631, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:51.086251 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.074642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:55949 (local address 127.23.44.62:38275)
0929 16:33:51.074763 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:51.074766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:51.075906 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 16:33:51.076737 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:51.076741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:51.076798 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:51.076897 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:51.077970 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.078502 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.079638 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.079991 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.080144 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.082462 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:51.082468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:51.082474 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:51.082506 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:51.084014 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:51.084611 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:51.084613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:51.084615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:51.084687 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:51.085105 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:51.085108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:51.085109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:51.085331 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:51.085407 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:51.086022 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 16:33:51.086136 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:33:51.087354 6857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.074977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:55949)
0929 16:33:51.075771 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:51.075808 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:51.076547 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:51.077058 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:51.077065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:51.077406 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:51.077829 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.077841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.078623 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.078627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.079488 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.079497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.080136 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.080140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.080276 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.080826 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:51.080841 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:51.082310 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:51.084128 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:51.084137 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:51.084152 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:51.084504 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:51.084774 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:51.084781 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:51.084786 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:51.084984 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:51.085432 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:51.085439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:51.085769 (+ 330us) client_negotiation.cc:769] Sending connection context
0929 16:33:51.085995 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 16:33:51.086334 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":503,"threads_started":1}
I20260929 16:33:51.088521 6855 heartbeater.cc:347] Connected to a master server at 127.23.44.62:38275
I20260929 16:33:51.088775 6855 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
I20260929 16:33:51.089231 6855 heartbeater.cc:511] Master 127.23.44.62:38275 requested a full tablet report, sending...
I20260929 16:33:51.090716 6664 ts_manager.cc:194] Registered new tserver with Master: b60e9e128c264e39b4ba808462bb7f20 (127.23.44.1:44873)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
I20260929 16:33:51.092123 6664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:55949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:33:51.106467 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38275
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:51.211181 6860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:51.211410 6860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:51.211452 6860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:51.215353 6860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:51.215399 6860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:51.215482 6860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:33:51.220417 6860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:51.221406 6860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:51.222319 6860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:51.231331 6868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:51.231302 6866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:51.231318 6865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:51.231403 6860 server_base.cc:1034] running on GCE node
I20260929 16:33:51.231920 6860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:51.232524 6860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:51.233692 6860 hybrid_clock.cc:648] HybridClock initialized: now 1790699631233679 us; error 26 us; skew 500 ppm
Sep 29 16:33:51 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699631, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:51.236250 6860 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:33:51.237038 6860 webserver.cc:460] Webserver started at http://127.23.44.2:36337/ using document root <none> and password file <none>
I20260929 16:33:51.237862 6860 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:51.238015 6860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:51.238255 6860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:51.240489 6860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "1a6d701c4f9243a98beaae639a8018b3"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "7009e9420d44e6ab31148f2d28205d0c"
server_key_iv: "b91d2c1af39ffff07692ade58f50718e"
server_key_version: "encryptionkey@0"
I20260929 16:33:51.241057 6860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "1a6d701c4f9243a98beaae639a8018b3"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "7009e9420d44e6ab31148f2d28205d0c"
server_key_iv: "b91d2c1af39ffff07692ade58f50718e"
server_key_version: "encryptionkey@0"
I20260929 16:33:51.244654 6860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:33:51.247067 6875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:51.248270 6860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:33:51.248411 6860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "1a6d701c4f9243a98beaae639a8018b3"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "7009e9420d44e6ab31148f2d28205d0c"
server_key_iv: "b91d2c1af39ffff07692ade58f50718e"
server_key_version: "encryptionkey@0"
I20260929 16:33:51.248521 6860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:51.274372 6860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:51.277498 6860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:51.277691 6860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:51.278249 6860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:51.279021 6860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:51.279116 6860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:51.279184 6860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:51.279229 6860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:51.287715 6860 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:43825
I20260929 16:33:51.287757 6988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:43825 every 8 connection(s)
I20260929 16:33:51.288525 6860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:51.291847 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 6860
I20260929 16:33:51.291990 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:33:51.292303 23728 external_mini_cluster.cc:1428] Setting key 5a23c368276ecc811b3ea507020a7726
Sep 29 16:33:51 dist-test-slave-mlmz krb5kdc[6619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699631, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:33:51.302019 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.290202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:53509 (local address 127.23.44.62:38275)
0929 16:33:51.290324 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:51.290327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:51.291063 (+ 736us) server_negotiation.cc:407] Connection header received
0929 16:33:51.292425 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:51.292429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:51.292488 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:51.292589 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:51.293871 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.294386 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.295467 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.295689 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.295793 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.298337 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:51.298341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:51.298344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:51.298374 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:51.299867 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:51.300387 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:51.300392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:51.300394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:51.300474 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:51.300805 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:51.300810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:51.300812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:51.300987 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:51.301197 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:51.301716 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 16:33:51.301847 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 16:33:51.302760 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.290497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:53509)
0929 16:33:51.290905 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:51.290944 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:51.292227 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:51.292756 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:51.292762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:51.293173 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:51.293745 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.293759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.294502 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.294506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.295350 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.295358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.295827 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.295831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.295993 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.296551 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:51.296567 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:51.298182 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:51.300007 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:51.300013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:51.300026 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:51.300286 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:51.300569 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:51.300573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:51.300575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:51.300714 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:51.301190 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:51.301196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:51.301474 (+ 278us) client_negotiation.cc:769] Sending connection context
0929 16:33:51.301658 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 16:33:51.301973 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
I20260929 16:33:51.303962 6989 heartbeater.cc:347] Connected to a master server at 127.23.44.62:38275
I20260929 16:33:51.304258 6989 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:51.304833 6989 heartbeater.cc:511] Master 127.23.44.62:38275 requested a full tablet report, sending...
I20260929 16:33:51.305932 6664 ts_manager.cc:194] Registered new tserver with Master: 1a6d701c4f9243a98beaae639a8018b3 (127.23.44.2:43825)
I20260929 16:33:51.306483 6664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:53509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:33:51.318279 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38275
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:51.424242 6994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:51.424499 6994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:51.424573 6994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:51.428555 6994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:51.428628 6994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:51.428714 6994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:33:51.433751 6994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:33:51.434675 6994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:51.435600 6994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:51.444737 7000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:51.444826 6994 server_base.cc:1034] running on GCE node
W20260929 16:33:51.444799 6999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:51.444738 7002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:51.445431 6994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:51.445986 6994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:51.447180 6994 hybrid_clock.cc:648] HybridClock initialized: now 1790699631447165 us; error 33 us; skew 500 ppm
Sep 29 16:33:51 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699631, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:51.449888 6994 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:33:51.450731 6994 webserver.cc:460] Webserver started at http://127.23.44.3:34763/ using document root <none> and password file <none>
I20260929 16:33:51.451258 6994 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:51.451337 6994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:51.451555 6994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:51.453518 6994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "e4f3998e4ad043118723b6ed9244dd2d"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "99b4219ea8ba11e1f8e4a5eebd4664e4"
server_key_iv: "6f6d2556d93afe4fd422076001971226"
server_key_version: "encryptionkey@0"
I20260929 16:33:51.454059 6994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "e4f3998e4ad043118723b6ed9244dd2d"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "99b4219ea8ba11e1f8e4a5eebd4664e4"
server_key_iv: "6f6d2556d93afe4fd422076001971226"
server_key_version: "encryptionkey@0"
I20260929 16:33:51.457667 6994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:33:51.459859 7009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:51.460991 6994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:33:51.461126 6994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "e4f3998e4ad043118723b6ed9244dd2d"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "99b4219ea8ba11e1f8e4a5eebd4664e4"
server_key_iv: "6f6d2556d93afe4fd422076001971226"
server_key_version: "encryptionkey@0"
I20260929 16:33:51.461232 6994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:51.485656 6994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:51.488545 6994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:51.488791 6994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:51.489418 6994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:51.490307 6994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:51.490373 6994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:51.490444 6994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:51.490487 6994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:51.499190 6994 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:42193
I20260929 16:33:51.499224 7122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:42193 every 8 connection(s)
I20260929 16:33:51.500062 6994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:33:51.503592 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 6994
I20260929 16:33:51.503722 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:33:51.503989 23728 external_mini_cluster.cc:1428] Setting key b39e0bb482903bcbd2ce8fc4976c4ece
Sep 29 16:33:51 dist-test-slave-mlmz krb5kdc[6619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699631, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:33:51.513312 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.501783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:49607 (local address 127.23.44.62:38275)
0929 16:33:51.501924 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:51.501929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:51.502750 (+ 821us) server_negotiation.cc:407] Connection header received
0929 16:33:51.504151 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:51.504155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:51.504210 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:51.504299 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:51.505663 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.506281 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.507379 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.507599 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.507706 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.510042 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:51.510045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:51.510047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:51.510070 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:51.511299 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:51.511774 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:51.511779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:51.511781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:51.511861 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:51.512196 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:51.512199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:51.512201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:51.512371 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:51.512474 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:51.512996 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 16:33:51.513125 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:33:51.514110 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.502203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:49607)
0929 16:33:51.502613 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:51.502653 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:51.503934 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:51.504522 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:51.504529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:51.504940 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:51.505523 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.505537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.506420 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.506424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.507255 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.507265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.507736 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.507740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.507902 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.508415 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:51.508429 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:51.509870 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:51.511419 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:51.511424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:51.511436 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:51.511674 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:51.511958 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:51.511962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:51.511964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:51.512096 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:51.512483 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:51.512491 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:51.512746 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 16:33:51.512972 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 16:33:51.513316 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260929 16:33:51.515197 7123 heartbeater.cc:347] Connected to a master server at 127.23.44.62:38275
I20260929 16:33:51.515394 7123 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:51.515795 7123 heartbeater.cc:511] Master 127.23.44.62:38275 requested a full tablet report, sending...
I20260929 16:33:51.516799 6664 ts_manager.cc:194] Registered new tserver with Master: e4f3998e4ad043118723b6ed9244dd2d (127.23.44.3:42193)
I20260929 16:33:51.517577 6664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:49607
I20260929 16:33:51.517778 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:33:51.526897 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.519862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.23.44.62:38275)
0929 16:33:51.519966 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:51.519969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:51.520152 (+ 183us) server_negotiation.cc:407] Connection header received
0929 16:33:51.520216 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:51.520218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:51.520255 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:51.520324 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:51.520922 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.521491 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.522308 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.522515 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.522596 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.523938 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:51.523940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:51.523942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:51.523962 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:51.525453 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:51.525821 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:51.525823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:51.525824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:51.525893 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:51.526179 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:51.526181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:51.526182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:51.526374 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:51.526480 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:51.526709 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 16:33:51.526793 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 16:33:51.529412 6664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:33:51.532249 6664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:33:51.546478 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.540417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43825 (local address 127.0.0.1:54202)
0929 16:33:51.541030 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:51.541075 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:51.541200 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:51.542167 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:51.542174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:51.542202 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:51.542832 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.542842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.543838 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.543842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.545006 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.545015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.546123 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.546130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.546232 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.546252 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 16:33:51.546319 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 16:33:51.546372 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":504,"spinlock_wait_cycles":384,"thread_start_us":88,"threads_started":1}
I20260929 16:33:51.546634 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.540550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.23.44.2:43825)
0929 16:33:51.541509 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:51.541521 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:51.541545 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:33:51.541626 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:51.541631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:51.541820 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:51.541968 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:51.542939 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.543707 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.545134 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.545969 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.546074 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.546219 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:51.546360 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 16:33:51.546451 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":118,"threads_started":1}
I20260929 16:33:51.546430 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.540555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:44873 (local address 127.0.0.1:57148)
0929 16:33:51.541398 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:51.541412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:51.541498 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:51.542187 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:51.542190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:51.542218 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:51.542377 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.542384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.543647 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.543650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.545038 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.545049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.545899 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.545902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.546002 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.546036 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:33:51.546137 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 16:33:51.546276 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":57,"spinlock_wait_cycles":896,"thread_start_us":85,"threads_started":1}
I20260929 16:33:51.546964 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.541177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57148 (local address 127.23.44.1:44873)
0929 16:33:51.541765 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:51.541780 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:51.541803 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:33:51.541877 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:51.541882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:51.542050 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:51.542208 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:51.542492 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.543509 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.545182 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.545780 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.545898 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.546036 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:51.546702 (+ 666us) server_negotiation.cc:299] Negotiation successful
0929 16:33:51.546803 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":159,"threads_started":1}
I20260929 16:33:51.546764 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.541155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:42193 (local address 127.0.0.1:47244)
0929 16:33:51.542161 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:51.542173 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:51.542263 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:51.542652 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:51.542655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:51.542672 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:51.542767 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.542776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.544072 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.544075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.545190 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.545197 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.546353 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.546356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.546439 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.546461 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:33:51.546530 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:33:51.546597 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":4,"spinlock_wait_cycles":9472,"thread_start_us":129,"threads_started":1}
I20260929 16:33:51.547148 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.541658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.23.44.3:42193)
0929 16:33:51.542067 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:51.542079 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:51.542343 (+ 264us) server_negotiation.cc:407] Connection header received
0929 16:33:51.542410 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:51.542414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:51.542530 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:51.542668 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:51.542898 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.543924 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.545315 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.546085 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.546179 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.546288 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:51.546849 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 16:33:51.546960 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":141,"threads_started":1}
I20260929 16:33:51.549856 6923 tablet_service.cc:1467] Processing CreateTablet for tablet 8e693e56881d416d8f14275bb179242b (DEFAULT_TABLE table=test-table [id=667618f853324755aa9b03a20ece9860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:51.549856 6789 tablet_service.cc:1467] Processing CreateTablet for tablet 8e693e56881d416d8f14275bb179242b (DEFAULT_TABLE table=test-table [id=667618f853324755aa9b03a20ece9860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:51.550937 6789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e693e56881d416d8f14275bb179242b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:51.550937 6923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e693e56881d416d8f14275bb179242b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:51.555452 7057 tablet_service.cc:1467] Processing CreateTablet for tablet 8e693e56881d416d8f14275bb179242b (DEFAULT_TABLE table=test-table [id=667618f853324755aa9b03a20ece9860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:51.556552 7057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e693e56881d416d8f14275bb179242b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:51.558342 7140 tablet_bootstrap.cc:492] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Bootstrap starting.
I20260929 16:33:51.559594 7141 tablet_bootstrap.cc:492] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3: Bootstrap starting.
I20260929 16:33:51.560662 7140 tablet_bootstrap.cc:654] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:51.561764 7141 tablet_bootstrap.cc:654] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:51.561997 7140 log.cc:824] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:51.562641 7141 log.cc:824] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:51.563993 7140 tablet_bootstrap.cc:492] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: No bootstrap required, opened a new log
I20260929 16:33:51.564188 7140 ts_tablet_manager.cc:1397] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:33:51.564450 7141 tablet_bootstrap.cc:492] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3: No bootstrap required, opened a new log
I20260929 16:33:51.564622 7141 ts_tablet_manager.cc:1397] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260929 16:33:51.565559 7143 tablet_bootstrap.cc:492] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d: Bootstrap starting.
I20260929 16:33:51.567155 7143 tablet_bootstrap.cc:654] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:51.568114 7143 log.cc:824] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:51.568001 7140 raft_consensus.cc:348] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } }
I20260929 16:33:51.568310 7140 raft_consensus.cc:374] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:51.568387 7140 raft_consensus.cc:729] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60e9e128c264e39b4ba808462bb7f20, State: Initialized, Role: FOLLOWER
I20260929 16:33:51.568261 7141 raft_consensus.cc:348] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } }
I20260929 16:33:51.568526 7141 raft_consensus.cc:374] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:51.568590 7141 raft_consensus.cc:729] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a6d701c4f9243a98beaae639a8018b3, State: Initialized, Role: FOLLOWER
I20260929 16:33:51.568889 7140 consensus_queue.cc:260] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } }
I20260929 16:33:51.569026 7141 consensus_queue.cc:260] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } }
I20260929 16:33:51.569536 6855 heartbeater.cc:503] Master 127.23.44.62:38275 was elected leader, sending a full tablet report...
I20260929 16:33:51.569777 6989 heartbeater.cc:503] Master 127.23.44.62:38275 was elected leader, sending a full tablet report...
I20260929 16:33:51.569859 7140 ts_tablet_manager.cc:1428] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:33:51.570189 7143 tablet_bootstrap.cc:492] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d: No bootstrap required, opened a new log
I20260929 16:33:51.570345 7143 ts_tablet_manager.cc:1397] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:33:51.571223 7141 ts_tablet_manager.cc:1428] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 16:33:51.574121 6856 tablet.cc:2369] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:33:51.574105 7143 raft_consensus.cc:348] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } }
I20260929 16:33:51.574446 7143 raft_consensus.cc:374] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:51.574488 7143 raft_consensus.cc:729] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f3998e4ad043118723b6ed9244dd2d, State: Initialized, Role: FOLLOWER
I20260929 16:33:51.574867 7143 consensus_queue.cc:260] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } }
I20260929 16:33:51.575484 7123 heartbeater.cc:503] Master 127.23.44.62:38275 was elected leader, sending a full tablet report...
I20260929 16:33:51.575646 7143 ts_tablet_manager.cc:1428] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 16:33:51.676472 7148 raft_consensus.cc:482] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:51.676661 7148 raft_consensus.cc:504] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } }
I20260929 16:33:51.677866 7148 leader_election.cc:288] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a6d701c4f9243a98beaae639a8018b3 (127.23.44.2:43825), b60e9e128c264e39b4ba808462bb7f20 (127.23.44.1:44873)
I20260929 16:33:51.681521 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.678131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:44873 (local address 127.23.44.3:44525)
0929 16:33:51.678334 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:51.678349 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:51.678454 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:51.678738 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:51.678741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:51.678759 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:51.678880 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.678886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.679603 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.679608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.680419 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.680428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.681062 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.681065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.681170 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.681197 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:33:51.681318 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 16:33:51.681388 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":24}
I20260929 16:33:51.681521 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.678260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:44525 (local address 127.23.44.1:44873)
0929 16:33:51.678384 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:51.678389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:51.678405 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:33:51.678595 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:51.678597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:51.678637 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:51.678722 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:51.678992 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.679485 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.680549 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.680940 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.681057 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.681116 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:51.681338 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 16:33:51.681402 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:33:51.682112 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.678183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43825 (local address 127.23.44.3:35271)
0929 16:33:51.678585 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:51.678604 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:51.678736 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:51.679026 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:51.679029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:51.679045 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:51.679175 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.679181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.679970 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.679973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.680860 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:51.680869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.681658 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.681663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:51.681794 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.681828 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:33:51.681904 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:33:51.681970 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":19,"thread_start_us":121,"threads_started":1}
I20260929 16:33:51.682123 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.678249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:35271 (local address 127.23.44.2:43825)
0929 16:33:51.678401 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:51.678406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:51.678709 (+ 303us) server_negotiation.cc:407] Connection header received
0929 16:33:51.678864 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:51.678867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:51.678917 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:51.679000 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:51.679296 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.679843 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.681006 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.681509 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.681638 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.681701 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:51.681922 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 16:33:51.681996 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:33:51.682312 6809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e693e56881d416d8f14275bb179242b" candidate_uuid: "e4f3998e4ad043118723b6ed9244dd2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60e9e128c264e39b4ba808462bb7f20" is_pre_election: true
I20260929 16:33:51.682612 6809 raft_consensus.cc:2454] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4f3998e4ad043118723b6ed9244dd2d in term 0.
I20260929 16:33:51.682722 6943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e693e56881d416d8f14275bb179242b" candidate_uuid: "e4f3998e4ad043118723b6ed9244dd2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a6d701c4f9243a98beaae639a8018b3" is_pre_election: true
I20260929 16:33:51.682979 6943 raft_consensus.cc:2454] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4f3998e4ad043118723b6ed9244dd2d in term 0.
I20260929 16:33:51.683130 7011 leader_election.cc:302] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b60e9e128c264e39b4ba808462bb7f20, e4f3998e4ad043118723b6ed9244dd2d; no voters:
I20260929 16:33:51.683421 7148 raft_consensus.cc:2781] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:33:51.683514 7148 raft_consensus.cc:482] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:33:51.683558 7148 raft_consensus.cc:3037] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:51.684541 7148 raft_consensus.cc:504] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } }
I20260929 16:33:51.684868 7148 leader_election.cc:288] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [CANDIDATE]: Term 1 election: Requested vote from peers 1a6d701c4f9243a98beaae639a8018b3 (127.23.44.2:43825), b60e9e128c264e39b4ba808462bb7f20 (127.23.44.1:44873)
I20260929 16:33:51.685223 6943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e693e56881d416d8f14275bb179242b" candidate_uuid: "e4f3998e4ad043118723b6ed9244dd2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a6d701c4f9243a98beaae639a8018b3"
I20260929 16:33:51.685360 6809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e693e56881d416d8f14275bb179242b" candidate_uuid: "e4f3998e4ad043118723b6ed9244dd2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60e9e128c264e39b4ba808462bb7f20"
I20260929 16:33:51.685396 6943 raft_consensus.cc:3037] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:51.685511 6809 raft_consensus.cc:3037] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:51.686409 6943 raft_consensus.cc:2454] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f3998e4ad043118723b6ed9244dd2d in term 1.
I20260929 16:33:51.686702 7012 leader_election.cc:302] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a6d701c4f9243a98beaae639a8018b3, e4f3998e4ad043118723b6ed9244dd2d; no voters:
I20260929 16:33:51.686836 6809 raft_consensus.cc:2454] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f3998e4ad043118723b6ed9244dd2d in term 1.
I20260929 16:33:51.686853 7148 raft_consensus.cc:2781] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:51.687074 7148 raft_consensus.cc:686] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 1 LEADER]: Becoming Leader. State: Replica: e4f3998e4ad043118723b6ed9244dd2d, State: Running, Role: LEADER
I20260929 16:33:51.687428 7148 consensus_queue.cc:237] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } }
I20260929 16:33:51.690649 6663 catalog_manager.cc:5495] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d reported cstate change: term changed from 0 to 1, leader changed from <none> to e4f3998e4ad043118723b6ed9244dd2d (127.23.44.3). New cstate: current_term: 1 leader_uuid: "e4f3998e4ad043118723b6ed9244dd2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } health_report { overall_health: HEALTHY } } }
I20260929 16:33:51.733328 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.729792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.23.44.3:42193)
0929 16:33:51.729914 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:51.729918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:51.729933 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:33:51.730063 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:51.730065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:51.730124 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:51.730239 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:33:51.730493 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.730992 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.731851 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:51.732076 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:51.732189 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:51.732492 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:33:51.732890 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:33:51.733027 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:51.733090 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 16:33:51.733146 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 16:33:51.738965 6809 raft_consensus.cc:1261] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Refusing update from remote peer e4f3998e4ad043118723b6ed9244dd2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:51.738976 6943 raft_consensus.cc:1261] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3 [term 1 FOLLOWER]: Refusing update from remote peer e4f3998e4ad043118723b6ed9244dd2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:51.739697 7150 consensus_queue.cc:1035] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:51.739903 7148 consensus_queue.cc:1035] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:51.749845 7154 mvcc.cc:205] Tried to move back new op lower bound from 7334705691595345920 to 7334705691394699264. Current Snapshot: MvccSnapshot[applied={T|T < 7334705691595345920 or (T in {7334705691595345920})}]
I20260929 16:33:51.757076 6663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:51.757248 6663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:51.759653 6663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Sending DeleteTablet for 3 replicas of tablet 8e693e56881d416d8f14275bb179242b
I20260929 16:33:51.760475 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 6726
I20260929 16:33:51.760489 6923 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e693e56881d416d8f14275bb179242b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:51 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:54202
I20260929 16:33:51.760535 6789 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e693e56881d416d8f14275bb179242b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:51 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:57148
I20260929 16:33:51.761108 7057 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e693e56881d416d8f14275bb179242b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:51 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:47244
I20260929 16:33:51.761579 7163 tablet_replica.cc:330] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d: stopping tablet replica
I20260929 16:33:51.761880 7163 raft_consensus.cc:2229] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:33:51.762212 7163 raft_consensus.cc:2258] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:51.762229 7162 tablet_replica.cc:330] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3: stopping tablet replica
I20260929 16:33:51.762638 7162 raft_consensus.cc:2229] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:51.762917 7162 raft_consensus.cc:2258] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:51.763661 7163 ts_tablet_manager.cc:1905] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:51.764348 7162 ts_tablet_manager.cc:1905] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:51.766805 7163 ts_tablet_manager.cc:1918] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:51.766948 7163 log.cc:1196] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/8e693e56881d416d8f14275bb179242b
I20260929 16:33:51.767335 7163 ts_tablet_manager.cc:1939] T 8e693e56881d416d8f14275bb179242b P e4f3998e4ad043118723b6ed9244dd2d: Deleting consensus metadata
I20260929 16:33:51.767879 7162 ts_tablet_manager.cc:1918] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:51.768005 7162 log.cc:1196] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/8e693e56881d416d8f14275bb179242b
I20260929 16:33:51.768325 7162 ts_tablet_manager.cc:1939] T 8e693e56881d416d8f14275bb179242b P 1a6d701c4f9243a98beaae639a8018b3: Deleting consensus metadata
I20260929 16:33:51.768545 6650 catalog_manager.cc:4841] TS e4f3998e4ad043118723b6ed9244dd2d (127.23.44.3:42193): tablet 8e693e56881d416d8f14275bb179242b (table test-table [id=667618f853324755aa9b03a20ece9860]) successfully deleted
I20260929 16:33:51.769490 6650 catalog_manager.cc:4841] TS 1a6d701c4f9243a98beaae639a8018b3 (127.23.44.2:43825): tablet 8e693e56881d416d8f14275bb179242b (table test-table [id=667618f853324755aa9b03a20ece9860]) successfully deleted
W20260929 16:33:51.769891 6649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:44873 (error 108)
I20260929 16:33:51.770674 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 6860
I20260929 16:33:51.770922 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:51.770607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:44873 (local address 127.0.0.1:57158)
0929 16:33:51.770779 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:51.770846 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 16:33:51.771492 6649 catalog_manager.cc:4584] TS b60e9e128c264e39b4ba808462bb7f20 (127.23.44.1:44873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e693e56881d416d8f14275bb179242b: Network error: Client connection negotiation failed: client connection to 127.23.44.1:44873: connect: Connection refused (error 111)
I20260929 16:33:51.776522 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 6994
I20260929 16:33:51.782403 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 6632
I20260929 16:34:11.789426 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:38275
--webserver_interface=127.23.44.62
--webserver_port=36799
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:38275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:11.893899 7165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:11.894126 7165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:11.894199 7165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:11.898154 7165 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:34:11.898201 7165 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:34:11.898217 7165 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:34:11.898231 7165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:11.898248 7165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:11.898289 7165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:11.898322 7165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:11.898350 7165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:11.903411 7165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:38275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:38275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=36799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:11.904342 7165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:11.905323 7165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:11.917470 7174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:11.917495 7171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:11.917493 7172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:11.917546 7165 server_base.cc:1034] running on GCE node
I20260929 16:34:11.918305 7165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:11.919441 7165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:11.920594 7165 hybrid_clock.cc:648] HybridClock initialized: now 1790699651920565 us; error 42 us; skew 500 ppm
Sep 29 16:34:11 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699651, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:11.928179 7165 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:34:11.929085 7165 webserver.cc:460] Webserver started at http://127.23.44.62:36799/ using document root <none> and password file <none>
I20260929 16:34:11.929677 7165 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:11.929777 7165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:11.933351 7165 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 16:34:11.935335 7181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:11.936457 7165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:11.936590 7165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "96713613bfee4618ac4b7b48a57622e8"
format_stamp: "Formatted at 2026-09-29 16:33:50 on dist-test-slave-mlmz"
server_key: "3c1517f1fb409450faa8c0dbc9758fd1"
server_key_iv: "f7b641f0d1cba1dcf3e23f092432e81a"
server_key_version: "encryptionkey@0"
I20260929 16:34:11.937181 7165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:11.957182 7165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:11.960224 7165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:11.960420 7165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:11.967772 7165 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:38275
I20260929 16:34:11.968185 7233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:38275 every 8 connection(s)
I20260929 16:34:11.968829 7165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:11.975427 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 7165
I20260929 16:34:11.975924 7234 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:34:11.976145 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:44873
--local_ip_for_outbound_sockets=127.23.44.1
--tserver_master_addrs=127.23.44.62:38275
--webserver_port=40051
--webserver_interface=127.23.44.1
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:34:11.978211 7234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Bootstrap starting.
I20260929 16:34:12.007645 7234 log.cc:824] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:12.016793 7234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:34:12.017155 7234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Bootstrap complete.
I20260929 16:34:12.021245 7234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } }
I20260929 16:34:12.021675 7234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96713613bfee4618ac4b7b48a57622e8, State: Initialized, Role: FOLLOWER
I20260929 16:34:12.022125 7234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } }
I20260929 16:34:12.022230 7234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:12.022327 7234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:12.022449 7234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:34:12.024183 7234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } }
I20260929 16:34:12.024509 7234 leader_election.cc:302] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96713613bfee4618ac4b7b48a57622e8; no voters:
I20260929 16:34:12.024827 7234 leader_election.cc:288] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:34:12.024899 7238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:34:12.025121 7238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 2 LEADER]: Becoming Leader. State: Replica: 96713613bfee4618ac4b7b48a57622e8, State: Running, Role: LEADER
I20260929 16:34:12.025408 7238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } }
I20260929 16:34:12.026077 7234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:12.026261 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "96713613bfee4618ac4b7b48a57622e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } } }
I20260929 16:34:12.027626 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:12.027508 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96713613bfee4618ac4b7b48a57622e8. Latest consensus state: current_term: 2 leader_uuid: "96713613bfee4618ac4b7b48a57622e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } } }
I20260929 16:34:12.027834 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:12.031042 7253 catalog_manager.cc:1259] Loaded cluster ID: 4f8968dc40864761b79e657f53ebfd33
I20260929 16:34:12.031101 7253 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: loading cluster ID for follower catalog manager: success
I20260929 16:34:12.033538 7253 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: acquiring CA information for follower catalog manager: success
I20260929 16:34:12.035012 7253 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:34:12.035144 7249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:12.036966 7249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:12.037442 7249 catalog_manager.cc:1259] Loaded cluster ID: 4f8968dc40864761b79e657f53ebfd33
I20260929 16:34:12.037513 7249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:12.038069 7249 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:12.038478 7249 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Loaded TSK: 0
I20260929 16:34:12.054422 7249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Generated new TSK 1
I20260929 16:34:12.054652 7249 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:34:12.099982 7236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:12.100193 7236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:12.100232 7236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:12.104259 7236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:12.104311 7236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:12.104414 7236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:34:12.109431 7236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:44873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=40051
--tserver_master_addrs=127.23.44.62:38275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:12.110316 7236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:12.111290 7236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:12.121155 7262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:12.121332 7236 server_base.cc:1034] running on GCE node
W20260929 16:34:12.121176 7261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:12.121600 7264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:12.121971 7236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:12.122547 7236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:12.123726 7236 hybrid_clock.cc:648] HybridClock initialized: now 1790699652123704 us; error 40 us; skew 500 ppm
Sep 29 16:34:12 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699652, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:12.126469 7236 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:34:12.127364 7236 webserver.cc:460] Webserver started at http://127.23.44.1:40051/ using document root <none> and password file <none>
I20260929 16:34:12.127897 7236 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:12.127996 7236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:12.131569 7236 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:34:12.133584 7271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:12.134763 7236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:34:12.134902 7236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "b60e9e128c264e39b4ba808462bb7f20"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "20ecd2c0a8e5bfe5d6d59e4e0199ad0f"
server_key_iv: "f400adbdb855c06d56d9b491de81ab56"
server_key_version: "encryptionkey@0"
I20260929 16:34:12.135401 7236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:12.146131 7236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:12.148873 7236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:12.149056 7236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:12.149633 7236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:12.150676 7278 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:34:12.154311 7236 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:34:12.154368 7236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 16:34:12.154409 7236 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:34:12.155915 7236 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:34:12.155970 7236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 16:34:12.156199 7278 tablet_bootstrap.cc:492] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Bootstrap starting.
I20260929 16:34:12.166266 7236 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:44873
I20260929 16:34:12.166522 7385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:44873 every 8 connection(s)
I20260929 16:34:12.167382 7236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:34:12.173175 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 7236
I20260929 16:34:12.173688 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:43825
--local_ip_for_outbound_sockets=127.23.44.2
--tserver_master_addrs=127.23.44.62:38275
--webserver_port=36337
--webserver_interface=127.23.44.2
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:34:12.177030 7278 log.cc:824] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:34:12 dist-test-slave-mlmz krb5kdc[6619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699652, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:12.185432 7278 tablet_bootstrap.cc:492] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:34:12.185874 7278 tablet_bootstrap.cc:492] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Bootstrap complete.
I20260929 16:34:12.187214 7278 ts_tablet_manager.cc:1397] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.004s
I20260929 16:34:12.187912 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.169536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:46955)
0929 16:34:12.170218 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:12.170276 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:12.171336 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:12.172739 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:12.172748 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:12.174300 (+ 1552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:12.174996 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.175015 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.176968 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.176974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.178340 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.178350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.179476 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.179479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.179665 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.180331 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:12.180351 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:12.182130 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:12.184559 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:12.184564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:12.184574 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:12.184906 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:12.185327 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:12.185330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:12.185332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:12.185491 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:12.186220 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:12.186229 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:12.186531 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 16:34:12.186700 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 16:34:12.187020 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":130,"threads_started":1}
I20260929 16:34:12.189494 7388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.169528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:46955 (local address 127.23.44.62:38275)
0929 16:34:12.169905 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:12.169925 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:12.170401 (+ 476us) server_negotiation.cc:407] Connection header received
0929 16:34:12.171853 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:12.171871 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:12.172285 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:12.172644 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:12.175220 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.176791 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.178481 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.179110 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.179269 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.182365 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:12.182373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:12.182388 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:12.182418 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:12.184431 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:12.185031 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:12.185036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:12.185042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:12.185182 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:12.185603 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:12.185608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:12.185609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:12.186042 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:12.186169 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:12.188451 (+ 2282us) server_negotiation.cc:299] Negotiation successful
0929 16:34:12.188711 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":115,"threads_started":1}
I20260929 16:34:12.190764 7278 raft_consensus.cc:348] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } }
I20260929 16:34:12.191535 7278 raft_consensus.cc:729] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60e9e128c264e39b4ba808462bb7f20, State: Initialized, Role: FOLLOWER
I20260929 16:34:12.191579 7386 heartbeater.cc:347] Connected to a master server at 127.23.44.62:38275
I20260929 16:34:12.191768 7386 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:12.192041 7278 consensus_queue.cc:260] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } }
I20260929 16:34:12.192255 7386 heartbeater.cc:511] Master 127.23.44.62:38275 requested a full tablet report, sending...
I20260929 16:34:12.192952 7278 ts_tablet_manager.cc:1428] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:34:12.194552 7198 ts_manager.cc:194] Registered new tserver with Master: b60e9e128c264e39b4ba808462bb7f20 (127.23.44.1:44873)
I20260929 16:34:12.198064 7198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:46955
I20260929 16:34:12.200588 7386 heartbeater.cc:503] Master 127.23.44.62:38275 was elected leader, sending a full tablet report...
Sep 29 16:34:12 dist-test-slave-mlmz krb5kdc[6619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699651, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:34:12.208142 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.197017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:44873 (local address 127.0.0.1:40968)
0929 16:34:12.197915 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:12.197960 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:12.198072 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:12.198838 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:12.198843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:12.199242 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:12.199372 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.199380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.200986 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.200989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.202020 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.202030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.202638 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.202642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.202743 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.203311 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:12.203323 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:12.204619 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:12.206491 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:12.206500 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:12.206505 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:12.206730 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:12.207033 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:12.207036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:12.207038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:12.207125 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:12.207548 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:12.207554 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:12.207667 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 16:34:12.207797 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 16:34:12.207976 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":98,"threads_started":1}
I20260929 16:34:12.208163 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.197712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.23.44.1:44873)
0929 16:34:12.198402 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:12.198413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:12.198433 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:34:12.198491 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:12.198495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:12.198678 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:12.199516 (+ 838us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:12.199831 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.200858 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.202151 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.202515 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.202638 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.204756 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:12.204765 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:12.204782 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:12.204808 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:12.206365 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:12.206845 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:12.206850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:12.206852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:12.206933 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:12.207222 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:12.207225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:12.207227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:12.207428 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:12.207559 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:12.207891 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 16:34:12.208020 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":102,"threads_started":1}
I20260929 16:34:12.209383 7318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e693e56881d416d8f14275bb179242b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:51 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:40968
I20260929 16:34:12.210206 7395 tablet_replica.cc:330] stopping tablet replica
I20260929 16:34:12.210489 7395 raft_consensus.cc:2229] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:34:12.210721 7395 raft_consensus.cc:2258] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:12.211840 7395 ts_tablet_manager.cc:1905] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:12.215345 7395 ts_tablet_manager.cc:1918] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:12.215480 7395 log.cc:1196] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal/wals/8e693e56881d416d8f14275bb179242b
I20260929 16:34:12.215749 7395 ts_tablet_manager.cc:1939] T 8e693e56881d416d8f14275bb179242b P b60e9e128c264e39b4ba808462bb7f20: Deleting consensus metadata
I20260929 16:34:12.217325 7183 catalog_manager.cc:4841] TS b60e9e128c264e39b4ba808462bb7f20 (127.23.44.1:44873): tablet 8e693e56881d416d8f14275bb179242b (table test-table [id=667618f853324755aa9b03a20ece9860]) successfully deleted
W20260929 16:34:12.294761 7390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:12.294991 7390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:12.295029 7390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:12.298924 7390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:12.298971 7390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:12.299026 7390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:34:12.304056 7390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:43825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=36337
--tserver_master_addrs=127.23.44.62:38275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:12.304982 7390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:12.305938 7390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:12.314795 7403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:12.314790 7400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:12.314790 7401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:12.315157 7390 server_base.cc:1034] running on GCE node
I20260929 16:34:12.315537 7390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:12.316169 7390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:12.317353 7390 hybrid_clock.cc:648] HybridClock initialized: now 1790699652317329 us; error 30 us; skew 500 ppm
Sep 29 16:34:12 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699652, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:12.320123 7390 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:34:12.320927 7390 webserver.cc:460] Webserver started at http://127.23.44.2:36337/ using document root <none> and password file <none>
I20260929 16:34:12.321514 7390 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:12.321620 7390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:12.325333 7390 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 16:34:12.327584 7410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:12.328898 7390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:34:12.329036 7390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "1a6d701c4f9243a98beaae639a8018b3"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "7009e9420d44e6ab31148f2d28205d0c"
server_key_iv: "b91d2c1af39ffff07692ade58f50718e"
server_key_version: "encryptionkey@0"
I20260929 16:34:12.329627 7390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:12.339569 7390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:12.342407 7390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:12.342588 7390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:12.343094 7390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:12.343824 7390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:12.343894 7390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:12.343957 7390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:12.344007 7390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:12.352596 7390 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:43825
I20260929 16:34:12.352615 7523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:43825 every 8 connection(s)
I20260929 16:34:12.353514 7390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:34:12.361624 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 7390
I20260929 16:34:12.362098 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:42193
--local_ip_for_outbound_sockets=127.23.44.3
--tserver_master_addrs=127.23.44.62:38275
--webserver_port=34763
--webserver_interface=127.23.44.3
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:34:12 dist-test-slave-mlmz krb5kdc[6619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699652, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:12.366778 7388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.355190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:54163 (local address 127.23.44.62:38275)
0929 16:34:12.355321 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:12.355325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:12.356109 (+ 784us) server_negotiation.cc:407] Connection header received
0929 16:34:12.357448 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:12.357451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:12.357497 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:12.357571 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:12.358638 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.359244 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.360320 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.360551 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.360644 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.363291 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:12.363294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:12.363296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:12.363322 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:12.364777 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:12.365299 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:12.365303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:12.365304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:12.365385 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:12.365721 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:12.365725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:12.365727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:12.365901 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:12.366006 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:12.366480 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 16:34:12.366607 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:34:12.367509 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.355553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:54163)
0929 16:34:12.355969 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:12.356006 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:12.357151 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:12.357719 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:12.357725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:12.358065 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:12.358490 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.358502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.359369 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.359372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.360197 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.360205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.360686 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.360689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.360816 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.361430 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:12.361452 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:12.363120 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:12.364913 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:12.364918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:12.364931 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:12.365176 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:12.365478 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:12.365483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:12.365484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:12.365616 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:12.366004 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:12.366011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:12.366239 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 16:34:12.366434 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 16:34:12.366739 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260929 16:34:12.368796 7524 heartbeater.cc:347] Connected to a master server at 127.23.44.62:38275
I20260929 16:34:12.369019 7524 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:12.369644 7524 heartbeater.cc:511] Master 127.23.44.62:38275 requested a full tablet report, sending...
I20260929 16:34:12.370776 7198 ts_manager.cc:194] Registered new tserver with Master: 1a6d701c4f9243a98beaae639a8018b3 (127.23.44.2:43825)
I20260929 16:34:12.371646 7198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:54163
W20260929 16:34:12.470207 7527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:12.470435 7527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:12.470475 7527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:12.474267 7527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:12.474311 7527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:12.474365 7527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:34:12.479319 7527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:42193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=34763
--tserver_master_addrs=127.23.44.62:38275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:12.480175 7527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:12.481086 7527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:12.489938 7532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:12.490001 7533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:12.489943 7535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:12.490150 7527 server_base.cc:1034] running on GCE node
I20260929 16:34:12.490615 7527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:12.491223 7527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:12.492389 7527 hybrid_clock.cc:648] HybridClock initialized: now 1790699652492367 us; error 36 us; skew 500 ppm
Sep 29 16:34:12 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699652, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:12.495286 7527 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:34:12.496112 7527 webserver.cc:460] Webserver started at http://127.23.44.3:34763/ using document root <none> and password file <none>
I20260929 16:34:12.496626 7527 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:12.496693 7527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:12.500152 7527 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:12.502140 7542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:12.503239 7527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:12.503378 7527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "e4f3998e4ad043118723b6ed9244dd2d"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "99b4219ea8ba11e1f8e4a5eebd4664e4"
server_key_iv: "6f6d2556d93afe4fd422076001971226"
server_key_version: "encryptionkey@0"
I20260929 16:34:12.503866 7527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:12.539063 7527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:12.542018 7527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:12.542204 7527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:12.542742 7527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:12.543509 7527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:12.543577 7527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:12.543644 7527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:12.543694 7527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:12.555269 7655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:42193 every 8 connection(s)
I20260929 16:34:12.555325 7527 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:42193
I20260929 16:34:12.556643 7527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:34:12.558220 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 7527
Sep 29 16:34:12 dist-test-slave-mlmz krb5kdc[6619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699652, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:12.567814 7388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.558898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.23.44.62:38275)
0929 16:34:12.559198 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:12.559202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:12.559264 (+ 62us) server_negotiation.cc:407] Connection header received
0929 16:34:12.559463 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:12.559467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:12.559525 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:12.559621 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:12.560492 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.561067 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.562066 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.562324 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.562447 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.564155 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:12.564158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:12.564160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:12.564181 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:12.566024 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:12.566438 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:12.566444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:12.566449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:12.566548 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:12.566905 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:12.566908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:12.566909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:12.567095 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:12.567321 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:12.567520 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 16:34:12.567629 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260929 16:34:12.571421 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.559898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:49025 (local address 127.23.44.62:38275)
0929 16:34:12.560189 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:12.560194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:12.560211 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:34:12.560975 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:12.560978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:12.561027 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:12.561135 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:12.562220 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.562733 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.564494 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.564812 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.564893 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.567701 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:12.567708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:12.567711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:12.567738 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:12.569515 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:12.570005 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:12.570009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:12.570011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:12.570097 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:12.570421 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:12.570426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:12.570428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:12.570592 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:12.570694 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:12.571143 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 16:34:12.571251 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":112,"threads_started":1}
I20260929 16:34:12.572134 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.559015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:49025)
0929 16:34:12.559447 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:12.559494 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:12.560774 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:12.561325 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:12.561333 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:12.561655 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:12.562084 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.562096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.562899 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.562902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.564380 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.564391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.564976 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.564984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.565237 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.565753 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:12.565780 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:12.567522 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:12.569625 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:12.569630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:12.569643 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:12.569902 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:12.570187 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:12.570192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:12.570194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:12.570327 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:12.570693 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:12.570699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:12.570928 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 16:34:12.571103 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 16:34:12.571391 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":121,"threads_started":1}
I20260929 16:34:12.573119 7656 heartbeater.cc:347] Connected to a master server at 127.23.44.62:38275
I20260929 16:34:12.573385 7656 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:12.573836 7656 heartbeater.cc:511] Master 127.23.44.62:38275 requested a full tablet report, sending...
I20260929 16:34:12.574733 7198 ts_manager.cc:194] Registered new tserver with Master: e4f3998e4ad043118723b6ed9244dd2d (127.23.44.3:42193)
I20260929 16:34:12.575230 7198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:49025
I20260929 16:34:12.581147 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:34:12.584571 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.581736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.23.44.62:38275)
0929 16:34:12.581844 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:12.581848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:12.582071 (+ 223us) server_negotiation.cc:407] Connection header received
0929 16:34:12.582141 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:12.582143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:12.582225 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:12.582320 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:12.582501 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.582966 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.583716 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.583919 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.583991 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.584213 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:12.584323 (+ 110us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 16:34:12.584494 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37078: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 16:34:12.584678 7660 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37078: token expired
W20260929 16:34:12.584956 7128 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.44.62:38275)
I20260929 16:34:12.591799 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.585323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.23.44.62:38275)
0929 16:34:12.585446 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:12.585449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:12.585462 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:34:12.585534 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:12.585538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:12.585578 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:12.585658 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:12.586208 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.586700 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.587475 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.587680 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.587791 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.588948 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:12.588951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:12.588954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:12.588983 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:12.590277 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:12.590677 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:12.590682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:12.590683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:12.590759 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:12.591032 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:12.591035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:12.591037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:12.591189 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:12.591322 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:12.591551 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 16:34:12.591655 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:34:12.593799 7198 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:34:12.596761 7198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:34:12.603679 7318 tablet_service.cc:1467] Processing CreateTablet for tablet e2721e92e32d4fd28a09a5c40c59bd48 (DEFAULT_TABLE table=test-table [id=042051d89899488e925e5726d01f2034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:12.604286 7318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2721e92e32d4fd28a09a5c40c59bd48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:12.606768 7278 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: Bootstrap starting.
I20260929 16:34:12.607658 7278 tablet_bootstrap.cc:654] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:12.608412 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.602568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43825 (local address 127.0.0.1:40384)
0929 16:34:12.602692 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:12.602707 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:12.602806 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:12.603658 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:12.603661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:12.603696 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:12.603780 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.603786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.605349 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.605352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.606558 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.606568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.607876 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.607881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.608023 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.608084 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 16:34:12.608172 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:34:12.608242 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":3,"spinlock_wait_cycles":3840}
I20260929 16:34:12.608604 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.602895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:42193 (local address 127.0.0.1:50122)
0929 16:34:12.603592 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:12.603606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:12.603695 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:12.604367 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:12.604370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:12.604386 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:12.604461 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.604466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.605813 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.605816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.606926 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.606935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.608126 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.608129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.608230 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.608252 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:34:12.608342 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:34:12.608410 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"spinlock_wait_cycles":4992,"thread_start_us":84,"threads_started":1}
I20260929 16:34:12.608605 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.603072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.23.44.3:42193)
0929 16:34:12.603449 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:12.603459 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:12.603804 (+ 345us) server_negotiation.cc:407] Connection header received
0929 16:34:12.603906 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:12.603911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:12.604083 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:12.604216 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:12.604579 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.605674 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.607050 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.607978 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.608075 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.608244 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:12.608369 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 16:34:12.608462 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":134,"threads_started":1}
I20260929 16:34:12.608959 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.602750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.23.44.2:43825)
0929 16:34:12.603147 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:12.603163 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:12.603189 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:34:12.603268 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:12.603274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:12.603498 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:12.603666 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:12.603899 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.605033 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.606893 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.607719 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.607908 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.608054 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:12.608752 (+ 698us) server_negotiation.cc:299] Negotiation successful
0929 16:34:12.608846 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":100,"threads_started":1}
I20260929 16:34:12.609160 7278 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: No bootstrap required, opened a new log
I20260929 16:34:12.609277 7278 ts_tablet_manager.cc:1397] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:34:12.609704 7278 raft_consensus.cc:348] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:12.609823 7278 raft_consensus.cc:374] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:12.609861 7278 raft_consensus.cc:729] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60e9e128c264e39b4ba808462bb7f20, State: Initialized, Role: FOLLOWER
I20260929 16:34:12.609979 7278 consensus_queue.cc:260] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:12.610239 7278 ts_tablet_manager.cc:1428] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:34:12.611465 7458 tablet_service.cc:1467] Processing CreateTablet for tablet e2721e92e32d4fd28a09a5c40c59bd48 (DEFAULT_TABLE table=test-table [id=042051d89899488e925e5726d01f2034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:12.611466 7586 tablet_service.cc:1467] Processing CreateTablet for tablet e2721e92e32d4fd28a09a5c40c59bd48 (DEFAULT_TABLE table=test-table [id=042051d89899488e925e5726d01f2034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:12.612397 7458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2721e92e32d4fd28a09a5c40c59bd48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:12.612425 7586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2721e92e32d4fd28a09a5c40c59bd48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:12.618765 7670 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: Bootstrap starting.
I20260929 16:34:12.619700 7669 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Bootstrap starting.
I20260929 16:34:12.621385 7670 tablet_bootstrap.cc:654] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:12.621711 7669 tablet_bootstrap.cc:654] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:12.622304 7670 log.cc:824] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:12.622920 7669 log.cc:824] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:12.624291 7670 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: No bootstrap required, opened a new log
I20260929 16:34:12.624472 7670 ts_tablet_manager.cc:1397] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:34:12.624723 7669 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: No bootstrap required, opened a new log
I20260929 16:34:12.624891 7669 ts_tablet_manager.cc:1397] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:34:12.628360 7670 raft_consensus.cc:348] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:12.628407 7669 raft_consensus.cc:348] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:12.628625 7669 raft_consensus.cc:374] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:12.628625 7670 raft_consensus.cc:374] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:12.628683 7669 raft_consensus.cc:729] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f3998e4ad043118723b6ed9244dd2d, State: Initialized, Role: FOLLOWER
I20260929 16:34:12.628683 7670 raft_consensus.cc:729] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a6d701c4f9243a98beaae639a8018b3, State: Initialized, Role: FOLLOWER
I20260929 16:34:12.629115 7670 consensus_queue.cc:260] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:12.629115 7669 consensus_queue.cc:260] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:12.629732 7524 heartbeater.cc:503] Master 127.23.44.62:38275 was elected leader, sending a full tablet report...
I20260929 16:34:12.630007 7656 heartbeater.cc:503] Master 127.23.44.62:38275 was elected leader, sending a full tablet report...
I20260929 16:34:12.630066 7670 ts_tablet_manager.cc:1428] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:34:12.630066 7669 ts_tablet_manager.cc:1428] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 16:34:12.668789 7387 tablet.cc:2369] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:34:12.715492 7673 raft_consensus.cc:482] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:34:12.715687 7673 raft_consensus.cc:504] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:12.716827 7673 leader_election.cc:288] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4f3998e4ad043118723b6ed9244dd2d (127.23.44.3:42193), b60e9e128c264e39b4ba808462bb7f20 (127.23.44.1:44873)
I20260929 16:34:12.720777 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.717100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:42193 (local address 127.23.44.2:45241)
0929 16:34:12.717344 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:12.717362 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:12.717460 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:12.717787 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:12.717790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:12.717831 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:12.717949 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.717955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.718806 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.718811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.719728 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.719737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.720349 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.720352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.720454 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.720482 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:34:12.720576 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 16:34:12.720642 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"spinlock_wait_cycles":27136}
I20260929 16:34:12.720777 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.717394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:45241 (local address 127.23.44.3:42193)
0929 16:34:12.717524 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:12.717529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:12.717546 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:34:12.717609 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:12.717611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:12.717672 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:12.717775 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:12.718064 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.718694 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.719841 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.720232 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.720349 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.720412 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:12.720596 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 16:34:12.720664 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:34:12.721302 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.717114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:44873 (local address 127.23.44.2:40359)
0929 16:34:12.717650 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:12.717661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:12.717838 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:12.718130 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:12.718132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:12.718149 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:12.718261 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.718265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.719072 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.719077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.720055 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:12.720063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.720722 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.720725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:12.720813 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.720968 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 16:34:12.721066 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 16:34:12.721130 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":203,"spinlock_wait_cycles":3200,"thread_start_us":100,"threads_started":1}
I20260929 16:34:12.721298 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.717186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:40359 (local address 127.23.44.1:44873)
0929 16:34:12.717475 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:12.717480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:12.717759 (+ 279us) server_negotiation.cc:407] Connection header received
0929 16:34:12.717943 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:12.717946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:12.718007 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:12.718110 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:12.718374 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.718969 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.720176 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.720599 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.720722 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.720892 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:12.721083 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 16:34:12.721151 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":104,"threads_started":1}
I20260929 16:34:12.721606 7608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2721e92e32d4fd28a09a5c40c59bd48" candidate_uuid: "1a6d701c4f9243a98beaae639a8018b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f3998e4ad043118723b6ed9244dd2d" is_pre_election: true
I20260929 16:34:12.721902 7608 raft_consensus.cc:2454] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a6d701c4f9243a98beaae639a8018b3 in term 0.
I20260929 16:34:12.721953 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2721e92e32d4fd28a09a5c40c59bd48" candidate_uuid: "1a6d701c4f9243a98beaae639a8018b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60e9e128c264e39b4ba808462bb7f20" is_pre_election: true
I20260929 16:34:12.722254 7340 raft_consensus.cc:2454] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a6d701c4f9243a98beaae639a8018b3 in term 0.
I20260929 16:34:12.722324 7413 leader_election.cc:302] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a6d701c4f9243a98beaae639a8018b3, e4f3998e4ad043118723b6ed9244dd2d; no voters:
I20260929 16:34:12.722630 7673 raft_consensus.cc:2781] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:34:12.722711 7673 raft_consensus.cc:482] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:34:12.722752 7673 raft_consensus.cc:3037] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:12.723740 7673 raft_consensus.cc:504] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:12.724044 7673 leader_election.cc:288] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [CANDIDATE]: Term 1 election: Requested vote from peers e4f3998e4ad043118723b6ed9244dd2d (127.23.44.3:42193), b60e9e128c264e39b4ba808462bb7f20 (127.23.44.1:44873)
I20260929 16:34:12.724341 7608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2721e92e32d4fd28a09a5c40c59bd48" candidate_uuid: "1a6d701c4f9243a98beaae639a8018b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f3998e4ad043118723b6ed9244dd2d"
I20260929 16:34:12.724341 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2721e92e32d4fd28a09a5c40c59bd48" candidate_uuid: "1a6d701c4f9243a98beaae639a8018b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60e9e128c264e39b4ba808462bb7f20"
I20260929 16:34:12.724457 7340 raft_consensus.cc:3037] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:12.724457 7608 raft_consensus.cc:3037] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:12.725682 7608 raft_consensus.cc:2454] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a6d701c4f9243a98beaae639a8018b3 in term 1.
I20260929 16:34:12.725685 7340 raft_consensus.cc:2454] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a6d701c4f9243a98beaae639a8018b3 in term 1.
I20260929 16:34:12.725958 7413 leader_election.cc:302] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a6d701c4f9243a98beaae639a8018b3, e4f3998e4ad043118723b6ed9244dd2d; no voters:
I20260929 16:34:12.726116 7673 raft_consensus.cc:2781] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:12.726300 7673 raft_consensus.cc:686] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 1 LEADER]: Becoming Leader. State: Replica: 1a6d701c4f9243a98beaae639a8018b3, State: Running, Role: LEADER
I20260929 16:34:12.726604 7673 consensus_queue.cc:237] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:12.729681 7196 catalog_manager.cc:5495] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a6d701c4f9243a98beaae639a8018b3 (127.23.44.2). New cstate: current_term: 1 leader_uuid: "1a6d701c4f9243a98beaae639a8018b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } health_report { overall_health: UNKNOWN } } }
I20260929 16:34:12.762205 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.758798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.23.44.2:43825)
0929 16:34:12.758954 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:12.758959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:12.758973 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:34:12.759136 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:12.759139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:12.759192 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:12.759304 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:12.759534 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.760054 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.760849 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:12.761065 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:12.761172 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:12.761480 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:12.761836 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:12.761963 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:12.762020 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 16:34:12.762070 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:34:12.767709 7608 raft_consensus.cc:1261] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [term 1 FOLLOWER]: Refusing update from remote peer 1a6d701c4f9243a98beaae639a8018b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:12.767737 7340 raft_consensus.cc:1261] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Refusing update from remote peer 1a6d701c4f9243a98beaae639a8018b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:12.768440 7677 consensus_queue.cc:1035] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:12.768606 7673 consensus_queue.cc:1035] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:12.773506 7680 mvcc.cc:205] Tried to move back new op lower bound from 7334705777729392640 to 7334705777571196928. Current Snapshot: MvccSnapshot[applied={T|T < 7334705777729392640}]
I20260929 16:34:12.774487 7681 mvcc.cc:205] Tried to move back new op lower bound from 7334705777729392640 to 7334705777571196928. Current Snapshot: MvccSnapshot[applied={T|T < 7334705777729392640}]
I20260929 16:34:12.782332 7198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:12.782495 7198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:12.784559 7198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Sending DeleteTablet for 3 replicas of tablet e2721e92e32d4fd28a09a5c40c59bd48
I20260929 16:34:12.785184 7318 tablet_service.cc:1514] Processing DeleteTablet for tablet e2721e92e32d4fd28a09a5c40c59bd48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:12 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:40968
I20260929 16:34:12.785288 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 7236
I20260929 16:34:12.785326 7586 tablet_service.cc:1514] Processing DeleteTablet for tablet e2721e92e32d4fd28a09a5c40c59bd48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:12 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:50122
I20260929 16:34:12.789700 7689 tablet_replica.cc:330] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: stopping tablet replica
I20260929 16:34:12.789674 7458 tablet_service.cc:1514] Processing DeleteTablet for tablet e2721e92e32d4fd28a09a5c40c59bd48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:12 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:40384
I20260929 16:34:12.790072 7689 raft_consensus.cc:2229] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:34:12.790118 7690 tablet_replica.cc:330] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: stopping tablet replica
I20260929 16:34:12.790378 7690 raft_consensus.cc:2229] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:34:12.790519 7689 raft_consensus.cc:2258] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:12.790686 7690 raft_consensus.cc:2258] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 16:34:12.791342 7183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:44873 (error 108)
I20260929 16:34:12.792245 7690 ts_tablet_manager.cc:1905] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:12.792246 7689 ts_tablet_manager.cc:1905] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:12.792364 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 7390
I20260929 16:34:12.795071 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.794787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:44873 (local address 127.0.0.1:40974)
0929 16:34:12.794924 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:12.794994 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:34:12.795513 7689 ts_tablet_manager.cc:1918] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:12.795600 7689 log.cc:1196] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/e2721e92e32d4fd28a09a5c40c59bd48
W20260929 16:34:12.795248 7183 catalog_manager.cc:4584] TS b60e9e128c264e39b4ba808462bb7f20 (127.23.44.1:44873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2721e92e32d4fd28a09a5c40c59bd48: Network error: Client connection negotiation failed: client connection to 127.23.44.1:44873: connect: Connection refused (error 111)
I20260929 16:34:12.795953 7689 ts_tablet_manager.cc:1939] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Deleting consensus metadata
I20260929 16:34:12.796931 7184 catalog_manager.cc:4841] TS e4f3998e4ad043118723b6ed9244dd2d (127.23.44.3:42193): tablet e2721e92e32d4fd28a09a5c40c59bd48 (table test-table [id=042051d89899488e925e5726d01f2034]) successfully deleted
I20260929 16:34:12.798126 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 7527
I20260929 16:34:12.799292 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:12.798279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43825 (local address 127.0.0.1:40406)
0929 16:34:12.799164 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:12.799230 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.2:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":820}
I20260929 16:34:12.803696 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 7165
I20260929 16:34:32.810029 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:38275
--webserver_interface=127.23.44.62
--webserver_port=36799
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:38275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:32.915943 7692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:32.916167 7692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:32.916235 7692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:32.920238 7692 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:34:32.920280 7692 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:34:32.920295 7692 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:34:32.920310 7692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:32.920362 7692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:32.920393 7692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:32.920439 7692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:32.920456 7692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:32.925400 7692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:38275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:38275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=36799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:32.926290 7692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:32.927208 7692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:32.936053 7700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:32.936028 7692 server_base.cc:1034] running on GCE node
W20260929 16:34:32.936046 7698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:32.936012 7697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:32.936839 7692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:32.937916 7692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:32.939085 7692 hybrid_clock.cc:648] HybridClock initialized: now 1790699672939059 us; error 39 us; skew 500 ppm
Sep 29 16:34:32 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699672, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:32.946254 7692 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:34:32.947127 7692 webserver.cc:460] Webserver started at http://127.23.44.62:36799/ using document root <none> and password file <none>
I20260929 16:34:32.947665 7692 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:32.947727 7692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:32.951277 7692 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:34:32.953364 7707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:32.954484 7692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:32.954581 7692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "96713613bfee4618ac4b7b48a57622e8"
format_stamp: "Formatted at 2026-09-29 16:33:50 on dist-test-slave-mlmz"
server_key: "3c1517f1fb409450faa8c0dbc9758fd1"
server_key_iv: "f7b641f0d1cba1dcf3e23f092432e81a"
server_key_version: "encryptionkey@0"
I20260929 16:34:32.955112 7692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:32.970857 7692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:32.973567 7692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:32.973748 7692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:32.980772 7692 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:38275
I20260929 16:34:32.981029 7759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:38275 every 8 connection(s)
I20260929 16:34:32.981853 7692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:32.985927 7760 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:34:32.986094 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 7692
I20260929 16:34:32.986514 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:44873
--local_ip_for_outbound_sockets=127.23.44.1
--tserver_master_addrs=127.23.44.62:38275
--webserver_port=40051
--webserver_interface=127.23.44.1
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:34:32.987617 7760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Bootstrap starting.
I20260929 16:34:33.004142 7760 log.cc:824] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:33.014598 7760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:34:33.014943 7760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Bootstrap complete.
I20260929 16:34:33.019022 7760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } }
I20260929 16:34:33.019446 7760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96713613bfee4618ac4b7b48a57622e8, State: Initialized, Role: FOLLOWER
I20260929 16:34:33.019893 7760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } }
I20260929 16:34:33.020040 7760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:33.020126 7760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:33.020251 7760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:34:33.022082 7760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } }
I20260929 16:34:33.022395 7760 leader_election.cc:302] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96713613bfee4618ac4b7b48a57622e8; no voters:
I20260929 16:34:33.022684 7760 leader_election.cc:288] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:34:33.022814 7764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:34:33.023056 7764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [term 3 LEADER]: Becoming Leader. State: Replica: 96713613bfee4618ac4b7b48a57622e8, State: Running, Role: LEADER
I20260929 16:34:33.023394 7764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } }
I20260929 16:34:33.023814 7760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:33.024430 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96713613bfee4618ac4b7b48a57622e8. Latest consensus state: current_term: 3 leader_uuid: "96713613bfee4618ac4b7b48a57622e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } } }
I20260929 16:34:33.024611 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:33.024842 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "96713613bfee4618ac4b7b48a57622e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96713613bfee4618ac4b7b48a57622e8" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38275 } } }
I20260929 16:34:33.025012 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:33.024996 7772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:33.030117 7772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:33.030771 7772 catalog_manager.cc:1259] Loaded cluster ID: 4f8968dc40864761b79e657f53ebfd33
I20260929 16:34:33.030829 7772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:33.033139 7772 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:33.033828 7772 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Loaded TSK: 0
I20260929 16:34:33.033905 7772 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 96713613bfee4618ac4b7b48a57622e8: Loaded TSK: 1
F20260929 16:34:33.034569 7772 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f005e0429ad google::LogMessage::Fail() at ??:0
@ 0x7f005e04588a google::LogMessage::SendToLog() at ??:0
@ 0x7f005e0424de google::LogMessage::Flush() at ??:0
@ 0x7f005e043a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0060d0618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0060cfef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f00689a77bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f006898199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0068982178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f00689c3df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f00689e282b std::function<>::operator()() at ??:0
@ 0x7f0068981b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0068982f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f006897f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f00689c3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f00602943d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0060294c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f00602963f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0060285512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f00618c96db start_thread at ??:0
@ 0x7f005cb4061f clone at ??:0
W20260929 16:34:33.116796 7762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:33.117000 7762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:33.117040 7762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:33.121688 7762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:33.121732 7762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:33.121788 7762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:34:33.127733 7762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:44873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=40051
--tserver_master_addrs=127.23.44.62:38275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:33.128634 7762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:33.129575 7762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:33.139038 7808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:33.139034 7807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:33.139042 7810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:33.139197 7762 server_base.cc:1034] running on GCE node
I20260929 16:34:33.139748 7762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:33.140374 7762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:33.141554 7762 hybrid_clock.cc:648] HybridClock initialized: now 1790699673141531 us; error 44 us; skew 500 ppm
Sep 29 16:34:33 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:33.144213 7762 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:34:33.145041 7762 webserver.cc:460] Webserver started at http://127.23.44.1:40051/ using document root <none> and password file <none>
I20260929 16:34:33.145625 7762 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:33.145684 7762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:33.149250 7762 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:34:33.151432 7817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:33.152653 7762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:33.152796 7762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "b60e9e128c264e39b4ba808462bb7f20"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "20ecd2c0a8e5bfe5d6d59e4e0199ad0f"
server_key_iv: "f400adbdb855c06d56d9b491de81ab56"
server_key_version: "encryptionkey@0"
I20260929 16:34:33.153337 7762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:33.168161 7762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:33.171239 7762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:33.171442 7762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:33.171989 7762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:33.173017 7824 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:34:33.178121 7762 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:34:33.178301 7762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 16:34:33.178378 7762 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:34:33.179903 7762 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:34:33.179975 7762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 16:34:33.180249 7824 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: Bootstrap starting.
I20260929 16:34:33.190421 7762 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:44873
I20260929 16:34:33.190599 7931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:44873 every 8 connection(s)
I20260929 16:34:33.191362 7762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:34:33.193979 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 7762
I20260929 16:34:33.194432 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:43825
--local_ip_for_outbound_sockets=127.23.44.2
--tserver_master_addrs=127.23.44.62:38275
--webserver_port=36337
--webserver_interface=127.23.44.2
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:34:33.201951 7824 log.cc:824] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:33.208796 7824 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:34:33.209178 7824 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: Bootstrap complete.
I20260929 16:34:33.210247 7824 ts_tablet_manager.cc:1397] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.011s
I20260929 16:34:33.212435 7824 raft_consensus.cc:348] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:33.212836 7824 raft_consensus.cc:729] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60e9e128c264e39b4ba808462bb7f20, State: Initialized, Role: FOLLOWER
I20260929 16:34:33.213392 7824 consensus_queue.cc:260] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:33.214273 7824 ts_tablet_manager.cc:1428] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260929 16:34:33.289130 7934 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:33.193379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:51119)
0929 16:34:33.194406 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:33.194461 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:33.195670 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:33.288402 (+ 92732us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":135,"threads_started":1}
W20260929 16:34:33.289743 7932 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:38275 (0 consecutive failures): Network error: Failed to ping master at 127.23.44.62:38275: Client connection negotiation failed: client connection to 127.23.44.62:38275: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:34:33.290460 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:33.290209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:45443)
0929 16:34:33.290336 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:33.290410 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:34:33.291152 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:33.290950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:43381)
0929 16:34:33.291067 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:33.291107 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 16:34:33.291393 7932 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:34:33.325547 7935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:33.325788 7935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:33.325858 7935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:33.329784 7935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:33.329850 7935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:33.329941 7935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:34:33.334949 7935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:43825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=36337
--tserver_master_addrs=127.23.44.62:38275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:33.335841 7935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:33.336732 7935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:33.345752 7945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:33.345803 7943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:33.345791 7935 server_base.cc:1034] running on GCE node
W20260929 16:34:33.345867 7942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:33.346350 7935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:33.346933 7935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:33.348116 7935 hybrid_clock.cc:648] HybridClock initialized: now 1790699673348094 us; error 37 us; skew 500 ppm
Sep 29 16:34:33 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:33.350693 7935 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:34:33.351532 7935 webserver.cc:460] Webserver started at http://127.23.44.2:36337/ using document root <none> and password file <none>
I20260929 16:34:33.352047 7935 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:33.352134 7935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:33.355600 7935 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:34:33.357694 7952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:33.358866 7935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:33.359009 7935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "1a6d701c4f9243a98beaae639a8018b3"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "7009e9420d44e6ab31148f2d28205d0c"
server_key_iv: "b91d2c1af39ffff07692ade58f50718e"
server_key_version: "encryptionkey@0"
I20260929 16:34:33.359515 7935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:33.371603 7935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:33.374178 7935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:33.374318 7935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:33.374778 7935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:33.375684 7959 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:34:33.379410 7935 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:34:33.379469 7935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 16:34:33.379591 7935 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:34:33.381150 7935 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:34:33.381191 7935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 16:34:33.381521 7959 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: Bootstrap starting.
I20260929 16:34:33.386466 7959 log.cc:824] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:33.391884 7935 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:43825
I20260929 16:34:33.391938 8067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:43825 every 8 connection(s)
I20260929 16:34:33.392666 7935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:34:33.394414 7959 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:34:33.394840 7959 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: Bootstrap complete.
I20260929 16:34:33.395159 7959 ts_tablet_manager.cc:1397] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260929 16:34:33.395857 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:33.394409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:35067)
0929 16:34:33.394935 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:33.395076 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":140,"threads_started":1}
W20260929 16:34:33.396232 8068 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:38275 (0 consecutive failures): Network error: Failed to ping master at 127.23.44.62:38275: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
I20260929 16:34:33.396772 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:33.396542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:55545)
0929 16:34:33.396660 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:33.396710 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 16:34:33.397413 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:33.397169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:38895)
0929 16:34:33.397302 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:33.397349 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:34:33.397573 8068 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:34:33.398418 7959 raft_consensus.cc:348] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:33.398785 7959 raft_consensus.cc:729] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a6d701c4f9243a98beaae639a8018b3, State: Initialized, Role: FOLLOWER
I20260929 16:34:33.399183 7959 consensus_queue.cc:260] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:33.400000 7959 ts_tablet_manager.cc:1428] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 16:34:33.400020 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:33.399814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:50379)
0929 16:34:33.399915 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:33.399963 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 16:34:33.400585 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:33.400419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:43873)
0929 16:34:33.400504 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:33.400542 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 16:34:33.401779 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 7935
I20260929 16:34:33.402174 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:42193
--local_ip_for_outbound_sockets=127.23.44.3
--tserver_master_addrs=127.23.44.62:38275
--webserver_port=34763
--webserver_interface=127.23.44.3
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:33.505687 8072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:33.505920 8072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:33.505961 8072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:33.509912 8072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:33.509960 8072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:33.510044 8072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:34:33.514994 8072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:42193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=34763
--tserver_master_addrs=127.23.44.62:38275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:33.515913 8072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:33.516837 8072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:33.525673 8077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:33.525754 8080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:33.525717 8078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:33.526005 8072 server_base.cc:1034] running on GCE node
I20260929 16:34:33.526461 8072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:33.527093 8072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:33.528306 8072 hybrid_clock.cc:648] HybridClock initialized: now 1790699673528289 us; error 31 us; skew 500 ppm
Sep 29 16:34:33 dist-test-slave-mlmz krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:33.530987 8072 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:34:33.531741 8072 webserver.cc:460] Webserver started at http://127.23.44.3:34763/ using document root <none> and password file <none>
I20260929 16:34:33.532265 8072 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:33.532354 8072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:33.535828 8072 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:33.537786 8087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:33.538879 8072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:33.539013 8072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "e4f3998e4ad043118723b6ed9244dd2d"
format_stamp: "Formatted at 2026-09-29 16:33:51 on dist-test-slave-mlmz"
server_key: "99b4219ea8ba11e1f8e4a5eebd4664e4"
server_key_iv: "6f6d2556d93afe4fd422076001971226"
server_key_version: "encryptionkey@0"
I20260929 16:34:33.539498 8072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:33.559518 8072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:33.562697 8072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:33.562887 8072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:33.563473 8072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:33.564239 8072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:33.564309 8072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:33.564378 8072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:33.564416 8072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:33.573299 8072 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:42193
I20260929 16:34:33.573333 8200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:42193 every 8 connection(s)
I20260929 16:34:33.574209 8072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:34:33.577759 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:33.576117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:47557)
0929 16:34:33.576548 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:33.576653 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":98,"threads_started":1}
I20260929 16:34:33.577956 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 8072
W20260929 16:34:33.578198 8201 heartbeater.cc:644] Failed to heartbeat to 127.23.44.62:38275 (0 consecutive failures): Network error: Failed to ping master at 127.23.44.62:38275: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
I20260929 16:34:33.578768 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:33.578523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:57933)
0929 16:34:33.578644 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:33.578697 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 16:34:33.579355 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:33.579182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:50259)
0929 16:34:33.579272 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:33.579310 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 16:34:33.579514 8201 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:34:34.292353 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:34.291938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:53513)
0929 16:34:34.292203 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:34.292257 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260929 16:34:34.401504 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:34.401126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:42275)
0929 16:34:34.401377 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:34.401427 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":60,"threads_started":1}
I20260929 16:34:34.580394 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:34.580007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:41351)
0929 16:34:34.580258 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:34.580306 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":64,"threads_started":1}
I20260929 16:34:34.815934 8208 raft_consensus.cc:482] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:34:34.816125 8208 raft_consensus.cc:504] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:34.817184 8208 leader_election.cc:288] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a6d701c4f9243a98beaae639a8018b3 (127.23.44.2:43825), e4f3998e4ad043118723b6ed9244dd2d (127.23.44.3:42193)
Sep 29 16:34:34 dist-test-slave-mlmz krb5kdc[6619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.2@KRBTEST.COM
Sep 29 16:34:34 dist-test-slave-mlmz krb5kdc[6619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.3@KRBTEST.COM
I20260929 16:34:34.830628 8212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:34.817961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:56721 (local address 127.23.44.3:42193)
0929 16:34:34.818264 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:34.818282 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:34.818315 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:34:34.818937 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:34.818946 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:34.819338 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:34.819618 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:34.820265 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:34.821236 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:34.822617 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:34.823078 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:34.823247 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:34.826262 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:34.826272 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:34.826282 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:34.826308 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:34.828034 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:34.828552 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:34.828557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:34.828562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:34.828682 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:34.829027 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:34.829033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:34.829036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:34.829465 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:34.829603 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:34.830150 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 16:34:34.830416 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
I20260929 16:34:34.830613 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:34.817362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43825 (local address 127.23.44.1:52977)
0929 16:34:34.817583 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:34.817596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:34.817739 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:34.819110 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:34.819116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:34.819601 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:34.820097 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:34.820116 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:34.821415 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:34.821418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:34.822472 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:34.822482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:34.823188 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:34.823191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:34.823401 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:34.824118 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:34.824130 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:34.825832 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:34.828105 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:34.828111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:34.828123 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:34.828440 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:34.828780 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:34.828784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:34.828786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:34.828921 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:34.829592 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:34.829598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:34.829853 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 16:34:34.830034 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 16:34:34.830342 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":237,"spinlock_wait_cycles":13056,"thread_start_us":78,"threads_started":1}
I20260929 16:34:34.830641 8210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:34.817559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:52977 (local address 127.23.44.2:43825)
0929 16:34:34.818012 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:34.818033 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:34.818079 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:34:34.818396 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:34.818401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:34.818758 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:34.819062 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:34.820256 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:34.821214 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:34.822589 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:34.823057 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:34.823225 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:34.826069 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:34.826074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:34.826086 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:34.826110 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:34.827973 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:34.828554 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:34.828559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:34.828571 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:34.828681 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:34.829032 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:34.829035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:34.829037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:34.829473 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:34.829613 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:34.830149 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 16:34:34.830390 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":90,"threads_started":1}
I20260929 16:34:34.830618 8211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:34.817703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:42193 (local address 127.23.44.1:56721)
0929 16:34:34.817992 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:34.818007 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:34.818095 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:34.819560 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:34.819563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:34.819774 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:34.820097 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:34.820113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:34.821403 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:34.821406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:34.822467 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:34.822476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:34.823202 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:34.823206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:34.823420 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:34.824073 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:34.824089 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:34.826079 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:34.828163 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:34.828169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:34.828171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:34.828431 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:34.828784 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:34.828788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:34.828790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:34.828930 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:34.829601 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:34.829604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:34.829853 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 16:34:34.830034 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 16:34:34.830429 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":318,"spinlock_wait_cycles":54144,"thread_start_us":60,"threads_started":1}
I20260929 16:34:34.831844 8022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2721e92e32d4fd28a09a5c40c59bd48" candidate_uuid: "b60e9e128c264e39b4ba808462bb7f20" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1a6d701c4f9243a98beaae639a8018b3" is_pre_election: true
I20260929 16:34:34.832151 8022 raft_consensus.cc:2454] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b60e9e128c264e39b4ba808462bb7f20 in term 1.
I20260929 16:34:34.833101 7820 leader_election.cc:302] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a6d701c4f9243a98beaae639a8018b3, b60e9e128c264e39b4ba808462bb7f20; no voters:
I20260929 16:34:34.833367 8208 raft_consensus.cc:2781] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 16:34:34.833457 8208 raft_consensus.cc:482] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:34:34.833516 8208 raft_consensus.cc:3037] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 1 FOLLOWER]: Advancing to term 2
W20260929 16:34:34.833460 7820 leader_election.cc:341] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4f3998e4ad043118723b6ed9244dd2d (127.23.44.3:42193): Not found: Tablet not found: e2721e92e32d4fd28a09a5c40c59bd48
I20260929 16:34:34.835454 8208 raft_consensus.cc:504] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:34.835790 8208 leader_election.cc:288] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [CANDIDATE]: Term 2 election: Requested vote from peers 1a6d701c4f9243a98beaae639a8018b3 (127.23.44.2:43825), e4f3998e4ad043118723b6ed9244dd2d (127.23.44.3:42193)
I20260929 16:34:34.836179 8022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2721e92e32d4fd28a09a5c40c59bd48" candidate_uuid: "b60e9e128c264e39b4ba808462bb7f20" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1a6d701c4f9243a98beaae639a8018b3"
I20260929 16:34:34.836299 8022 raft_consensus.cc:3037] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 1 FOLLOWER]: Advancing to term 2
W20260929 16:34:34.836555 7820 leader_election.cc:341] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer e4f3998e4ad043118723b6ed9244dd2d (127.23.44.3:42193): Not found: Tablet not found: e2721e92e32d4fd28a09a5c40c59bd48
I20260929 16:34:34.838284 8022 raft_consensus.cc:2454] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b60e9e128c264e39b4ba808462bb7f20 in term 2.
I20260929 16:34:34.838574 7820 leader_election.cc:302] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a6d701c4f9243a98beaae639a8018b3, b60e9e128c264e39b4ba808462bb7f20; no voters: e4f3998e4ad043118723b6ed9244dd2d
I20260929 16:34:34.838745 8208 raft_consensus.cc:2781] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:34:34.838900 8208 raft_consensus.cc:686] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [term 2 LEADER]: Becoming Leader. State: Replica: b60e9e128c264e39b4ba808462bb7f20, State: Running, Role: LEADER
I20260929 16:34:34.839210 8208 consensus_queue.cc:237] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:34.839576 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:34.839303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:44351)
0929 16:34:34.839454 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:34.839511 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 16:34:35.380748 8022 raft_consensus.cc:1261] T e2721e92e32d4fd28a09a5c40c59bd48 P 1a6d701c4f9243a98beaae639a8018b3 [term 2 FOLLOWER]: Refusing update from remote peer b60e9e128c264e39b4ba808462bb7f20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 16:34:35.381520 8215 consensus_queue.cc:1035] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 16:34:35.383392 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.383069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:55563)
0929 16:34:35.383276 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.383318 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":48,"threads_started":1}
I20260929 16:34:35.385670 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.385247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:42337)
0929 16:34:35.385506 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.385567 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
W20260929 16:34:35.385761 7820 consensus_peers.cc:487] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20 -> Peer e4f3998e4ad043118723b6ed9244dd2d (127.23.44.3:42193): Couldn't send request to peer e4f3998e4ad043118723b6ed9244dd2d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2721e92e32d4fd28a09a5c40c59bd48. This is attempt 1: this message will repeat every 5th retry.
I20260929 16:34:35.386082 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.385873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:39619)
0929 16:34:35.385988 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.386031 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 16:34:35.581629 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.581135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:43957)
0929 16:34:35.581474 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.581524 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":75,"threads_started":1}
I20260929 16:34:35.818217 8225 ts_tablet_manager.cc:927] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Initiating tablet copy from peer b60e9e128c264e39b4ba808462bb7f20 (127.23.44.1:44873)
I20260929 16:34:35.818884 8225 tablet_copy_client.cc:316] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: tablet copy: Beginning tablet copy session from remote peer at address 127.23.44.1:44873
Sep 29 16:34:35 dist-test-slave-mlmz krb5kdc[6619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.1@KRBTEST.COM
I20260929 16:34:35.828707 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.819363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:44873 (local address 127.23.44.3:43573)
0929 16:34:35.819507 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.819538 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:35.819649 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:35.820255 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:35.820261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:35.820601 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:35.820709 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:35.820715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:35.821747 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:35.821750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:35.822365 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:35.822373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:35.822955 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:35.822958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:35.823034 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:35.823545 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:35.823557 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:35.824941 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:35.826995 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:35.827003 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:35.827008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:35.827243 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:35.827554 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:35.827556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:35.827557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:35.827635 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:35.828105 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:35.828113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:35.828224 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 16:34:35.828378 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:34:35.828551 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 16:34:35.828774 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.819567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:43573 (local address 127.23.44.1:44873)
0929 16:34:35.819840 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:35.819852 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:35.819890 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:34:35.819934 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:35.819938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:35.820098 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:35.820210 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:35.820834 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:35.821644 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:35.822493 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:35.822836 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:35.822930 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:35.825068 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:35.825077 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:35.825087 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:35.825110 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:35.826886 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:35.827368 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:35.827375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:35.827377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:35.827465 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:35.827712 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:35.827714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:35.827715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:35.827982 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:35.828092 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:35.828504 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 16:34:35.828659 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":107,"threads_started":1}
I20260929 16:34:35.829770 7906 tablet_copy_service.cc:140] P b60e9e128c264e39b4ba808462bb7f20: Received BeginTabletCopySession request for tablet e2721e92e32d4fd28a09a5c40c59bd48 from peer e4f3998e4ad043118723b6ed9244dd2d ({username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:43573)
I20260929 16:34:35.829921 7906 tablet_copy_service.cc:161] P b60e9e128c264e39b4ba808462bb7f20: Beginning new tablet copy session on tablet e2721e92e32d4fd28a09a5c40c59bd48 from peer e4f3998e4ad043118723b6ed9244dd2d at {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:43573: session id = e4f3998e4ad043118723b6ed9244dd2d-e2721e92e32d4fd28a09a5c40c59bd48
I20260929 16:34:35.830960 7906 tablet_copy_source_session.cc:205] T e2721e92e32d4fd28a09a5c40c59bd48 P b60e9e128c264e39b4ba808462bb7f20: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:34:35.834587 8225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2721e92e32d4fd28a09a5c40c59bd48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:35.841888 8225 tablet_copy_client.cc:791] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: tablet copy: Starting download of 0 data blocks...
I20260929 16:34:35.842027 8225 tablet_copy_client.cc:655] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: tablet copy: Starting download of 1 WAL segments...
I20260929 16:34:35.843586 8225 tablet_copy_client.cc:531] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:34:35.845041 8225 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Bootstrap starting.
I20260929 16:34:35.849949 8225 log.cc:824] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:35.855703 8225 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:34:35.856002 8225 tablet_bootstrap.cc:492] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Bootstrap complete.
I20260929 16:34:35.856256 8225 ts_tablet_manager.cc:1397] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260929 16:34:35.858532 8225 raft_consensus.cc:348] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:35.858738 8225 raft_consensus.cc:729] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f3998e4ad043118723b6ed9244dd2d, State: Initialized, Role: FOLLOWER
I20260929 16:34:35.859130 8225 consensus_queue.cc:260] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6d701c4f9243a98beaae639a8018b3" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43825 } } peers { permanent_uuid: "e4f3998e4ad043118723b6ed9244dd2d" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 42193 } } peers { permanent_uuid: "b60e9e128c264e39b4ba808462bb7f20" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 44873 } }
I20260929 16:34:35.859930 8225 ts_tablet_manager.cc:1428] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260929 16:34:35.860247 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.859927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:46325)
0929 16:34:35.860099 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.860158 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 16:34:35.860625 7906 tablet_copy_service.cc:342] P b60e9e128c264e39b4ba808462bb7f20: Request end of tablet copy session e4f3998e4ad043118723b6ed9244dd2d-e2721e92e32d4fd28a09a5c40c59bd48 received from {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:43573
I20260929 16:34:35.860726 7906 tablet_copy_service.cc:433] P b60e9e128c264e39b4ba808462bb7f20: ending tablet copy session e4f3998e4ad043118723b6ed9244dd2d-e2721e92e32d4fd28a09a5c40c59bd48 on tablet e2721e92e32d4fd28a09a5c40c59bd48 with peer e4f3998e4ad043118723b6ed9244dd2d
I20260929 16:34:35.861156 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.860916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:53915)
0929 16:34:35.861031 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.861083 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 16:34:36.324875 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.324502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:40235)
0929 16:34:36.324729 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.324781 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260929 16:34:36.324906 8155 raft_consensus.cc:1203] T e2721e92e32d4fd28a09a5c40c59bd48 P e4f3998e4ad043118723b6ed9244dd2d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260929 16:34:36.325904 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.325684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:45453)
0929 16:34:36.325787 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.325844 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 16:34:36.387014 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.386640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:37251)
0929 16:34:36.386864 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.386915 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":59,"threads_started":1}
I20260929 16:34:37.326093 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:37.325670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:34795)
0929 16:34:37.325945 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:37.326000 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20260929 16:34:37.327059 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:37.326642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:58183)
0929 16:34:37.326923 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:37.326984 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":61,"threads_started":1}
I20260929 16:34:37.388098 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:37.387683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:56589)
0929 16:34:37.387960 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:37.388011 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260929 16:34:38.327394 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:38.326963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:55553)
0929 16:34:38.327214 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:38.327266 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
I20260929 16:34:38.328123 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:38.327724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:58807)
0929 16:34:38.327953 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:38.328017 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":53,"threads_started":1}
I20260929 16:34:38.328193 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:38.327969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:55513)
0929 16:34:38.328075 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:38.328117 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 16:34:38.389191 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:38.388741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:52935)
0929 16:34:38.389047 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:38.389101 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":65,"threads_started":1}
I20260929 16:34:39.329408 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:39.328960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:58247)
0929 16:34:39.329204 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:39.329296 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
I20260929 16:34:39.329504 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:39.329062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:50229)
0929 16:34:39.329325 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:39.329391 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":101,"threads_started":1}
I20260929 16:34:39.390512 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:39.390057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:56941)
0929 16:34:39.390328 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:39.390386 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":72,"threads_started":1}
I20260929 16:34:40.330442 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.330024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:51361)
0929 16:34:40.330261 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.330331 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260929 16:34:40.330642 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.330218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:59901)
0929 16:34:40.330464 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.330542 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260929 16:34:40.391649 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.391221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:40357)
0929 16:34:40.391499 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.391555 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
I20260929 16:34:41.331537 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.331086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:48609)
0929 16:34:41.331393 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.331445 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":105,"threads_started":1}
I20260929 16:34:41.331717 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.331302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:48285)
0929 16:34:41.331541 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.331613 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260929 16:34:41.392827 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.392389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:52751)
0929 16:34:41.392665 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.392745 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260929 16:34:42.332754 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.332307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:52871)
0929 16:34:42.332577 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:42.332639 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260929 16:34:42.332898 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.332511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:51791)
0929 16:34:42.332742 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:42.332792 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260929 16:34:42.393889 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.393522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:33485)
0929 16:34:42.393766 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:42.393813 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":101,"threads_started":1}
I20260929 16:34:43.334043 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:43.333647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:39031)
0929 16:34:43.333904 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:43.333955 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260929 16:34:43.334046 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:43.333665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:58711)
0929 16:34:43.333905 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:43.333963 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":115,"threads_started":1}
I20260929 16:34:43.394846 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:43.394477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:48741)
0929 16:34:43.394711 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:43.394762 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260929 16:34:44.335311 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:44.334913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:42487)
0929 16:34:44.335167 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:44.335219 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
I20260929 16:34:44.335455 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:44.334915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:49273)
0929 16:34:44.335279 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:44.335329 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":105,"threads_started":1}
I20260929 16:34:44.395905 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:44.395531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:47977)
0929 16:34:44.395778 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:44.395828 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":96,"threads_started":1}
I20260929 16:34:45.336515 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:45.336050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:52897)
0929 16:34:45.336321 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:45.336385 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260929 16:34:45.336748 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:45.336238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:55065)
0929 16:34:45.336541 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:45.336636 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":77,"threads_started":1}
I20260929 16:34:45.396912 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:45.396544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:33331)
0929 16:34:45.396779 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:45.396827 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":89,"threads_started":1}
I20260929 16:34:46.337752 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:46.337256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:46521)
0929 16:34:46.337565 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:46.337633 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":74,"threads_started":1}
I20260929 16:34:46.337910 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:46.337521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:48647)
0929 16:34:46.337738 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:46.337786 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260929 16:34:46.398043 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:46.397611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:46205)
0929 16:34:46.397854 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:46.397926 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":66,"threads_started":1}
I20260929 16:34:47.338975 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:47.338502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:35677)
0929 16:34:47.338775 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:47.338845 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
I20260929 16:34:47.339016 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:47.338583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:49805)
0929 16:34:47.338859 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:47.338928 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
I20260929 16:34:47.399045 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:47.398665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:48175)
0929 16:34:47.398920 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:47.398969 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
I20260929 16:34:48.340468 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:48.339960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.3:56331)
0929 16:34:48.340282 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:48.340351 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":85,"threads_started":1}
I20260929 16:34:48.340528 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:48.339960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.1:49181)
0929 16:34:48.340335 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:48.340414 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":88,"threads_started":1}
I20260929 16:34:48.400213 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:48.399808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38275 (local address 127.23.44.2:41673)
0929 16:34:48.400079 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:48.400131 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.62:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20260929 16:34:48.586649 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 7762
I20260929 16:34:48.592358 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 7935
I20260929 16:34:48.597700 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 8072
2026-09-29T16:34:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59115 ms)
[----------] 30 tests from SecurityITest (238166 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:48 dist-test-slave-mlmz krb5kdc[8272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:48 dist-test-slave-mlmz krb5kdc[8272](info): set up 2 sockets
Sep 29 16:34:48 dist-test-slave-mlmz krb5kdc[8272](info): commencing operation
krb5kdc: starting...
W20260929 16:34:49.625103 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:34:49 dist-test-slave-mlmz krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:34:49.743734 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:41661
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:41661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:49.847229 8285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:49.847492 8285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:49.847565 8285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:49.851493 8285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:49.851574 8285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:49.851613 8285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:49.851665 8285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:49.851702 8285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:49.856649 8285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:41661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:41661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:49.857582 8285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:49.858489 8285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:49.866534 8290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:49.866552 8291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:49.866514 8293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:49.866777 8285 server_base.cc:1034] running on GCE node
I20260929 16:34:49.867308 8285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:49.868309 8285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:49.869488 8285 hybrid_clock.cc:648] HybridClock initialized: now 1790699689869462 us; error 40 us; skew 500 ppm
Sep 29 16:34:49 dist-test-slave-mlmz krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699689, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:49.876453 8285 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:34:49.877393 8285 webserver.cc:460] Webserver started at http://127.23.44.62:40301/ using document root <none> and password file <none>
I20260929 16:34:49.877929 8285 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:49.878016 8285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:49.878233 8285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:49.880134 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "92344a3ff54e4e8ab73963750ec091fa"
format_stamp: "Formatted at 2026-09-29 16:34:49 on dist-test-slave-mlmz"
server_key: "3c4842f7cf13759cbbc2e44b93eed85f"
server_key_iv: "8d3fdc184902ae87ddd7f18821e96b78"
server_key_version: "encryptionkey@0"
I20260929 16:34:49.880685 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "92344a3ff54e4e8ab73963750ec091fa"
format_stamp: "Formatted at 2026-09-29 16:34:49 on dist-test-slave-mlmz"
server_key: "3c4842f7cf13759cbbc2e44b93eed85f"
server_key_iv: "8d3fdc184902ae87ddd7f18821e96b78"
server_key_version: "encryptionkey@0"
I20260929 16:34:49.884133 8285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 16:34:49.886515 8300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:49.887634 8285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:34:49.887769 8285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "92344a3ff54e4e8ab73963750ec091fa"
format_stamp: "Formatted at 2026-09-29 16:34:49 on dist-test-slave-mlmz"
server_key: "3c4842f7cf13759cbbc2e44b93eed85f"
server_key_iv: "8d3fdc184902ae87ddd7f18821e96b78"
server_key_version: "encryptionkey@0"
I20260929 16:34:49.887877 8285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:49.913378 8285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:49.916129 8285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:49.916333 8285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:49.923401 8285 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:41661
I20260929 16:34:49.923413 8352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:41661 every 8 connection(s)
I20260929 16:34:49.924274 8285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:49.927067 8353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:49.929044 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 8285
I20260929 16:34:49.929188 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:34:49.929508 23728 external_mini_cluster.cc:1428] Setting key 166268dde5395fb691e8ce61b9c4f275
I20260929 16:34:49.933597 8353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa: Bootstrap starting.
I20260929 16:34:49.935949 8353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:49.936803 8353 log.cc:824] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:34:49 dist-test-slave-mlmz krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:49.938680 8353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa: No bootstrap required, opened a new log
I20260929 16:34:49.941841 8353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92344a3ff54e4e8ab73963750ec091fa" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 41661 } }
I20260929 16:34:49.942068 8353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:49.942158 8353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92344a3ff54e4e8ab73963750ec091fa, State: Initialized, Role: FOLLOWER
I20260929 16:34:49.942602 8353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92344a3ff54e4e8ab73963750ec091fa" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 41661 } }
I20260929 16:34:49.942719 8353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:49.942880 8353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:49.943038 8353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:49.943427 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:49.930785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.23.44.62:41661)
0929 16:34:49.931168 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:49.931183 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:49.931215 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:34:49.931818 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:49.931832 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:49.932123 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:49.932368 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:49.933223 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:49.934256 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:49.935414 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:49.935852 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:49.935966 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:49.938700 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:49.938704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:49.938747 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:49.938779 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:49.940444 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:49.940898 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:49.940903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:49.940907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:49.941022 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:49.941324 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:49.941329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:49.941331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:49.941677 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:49.941893 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:49.942150 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 16:34:49.942559 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":133,"threads_started":1}
I20260929 16:34:49.944021 8353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92344a3ff54e4e8ab73963750ec091fa" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 41661 } }
I20260929 16:34:49.944370 8353 leader_election.cc:302] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92344a3ff54e4e8ab73963750ec091fa; no voters:
I20260929 16:34:49.944674 8353 leader_election.cc:288] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:49.944787 8358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:49.944983 8358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [term 1 LEADER]: Becoming Leader. State: Replica: 92344a3ff54e4e8ab73963750ec091fa, State: Running, Role: LEADER
I20260929 16:34:49.945330 8358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92344a3ff54e4e8ab73963750ec091fa" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 41661 } }
I20260929 16:34:49.946017 8353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:49.946420 8358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92344a3ff54e4e8ab73963750ec091fa. Latest consensus state: current_term: 1 leader_uuid: "92344a3ff54e4e8ab73963750ec091fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92344a3ff54e4e8ab73963750ec091fa" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 41661 } } }
I20260929 16:34:49.946604 8358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:49.946873 8359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92344a3ff54e4e8ab73963750ec091fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92344a3ff54e4e8ab73963750ec091fa" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 41661 } } }
I20260929 16:34:49.947075 8359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:49.947013 8362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:49.950496 8362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:49.955914 8362 catalog_manager.cc:1347] Generated new cluster ID: 166ef2b8ebc6477e85a2ef6fbb7e0ae2
I20260929 16:34:49.955976 8362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:49.971041 8362 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:49.972779 8362 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:49.979460 8362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa: Generated new TSK 0
I20260929 16:34:49.980139 8362 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:34:50.013137 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:41661
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:50.119468 8379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:50.119675 8379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:50.119714 8379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:50.123677 8379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:50.123723 8379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:50.123814 8379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:34:50.128791 8379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:41661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:50.129698 8379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:50.130620 8379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:50.139526 8387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:50.139490 8384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:50.139552 8385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:50.139732 8379 server_base.cc:1034] running on GCE node
I20260929 16:34:50.140165 8379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:50.140705 8379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:50.141860 8379 hybrid_clock.cc:648] HybridClock initialized: now 1790699690141853 us; error 29 us; skew 500 ppm
Sep 29 16:34:50 dist-test-slave-mlmz krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:50.144299 8379 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:34:50.145109 8379 webserver.cc:460] Webserver started at http://127.23.44.1:46849/ using document root <none> and password file <none>
I20260929 16:34:50.145728 8379 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:50.145813 8379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:50.146046 8379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:50.148097 8379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "89e1f14f3b4643db99b18ae75a45a5f5"
format_stamp: "Formatted at 2026-09-29 16:34:50 on dist-test-slave-mlmz"
server_key: "01cd9cbc7fbff48bb98fcadb5456d370"
server_key_iv: "a413e2d4455e6fb24dc2ba80f08fae27"
server_key_version: "encryptionkey@0"
I20260929 16:34:50.148665 8379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "89e1f14f3b4643db99b18ae75a45a5f5"
format_stamp: "Formatted at 2026-09-29 16:34:50 on dist-test-slave-mlmz"
server_key: "01cd9cbc7fbff48bb98fcadb5456d370"
server_key_iv: "a413e2d4455e6fb24dc2ba80f08fae27"
server_key_version: "encryptionkey@0"
I20260929 16:34:50.151997 8379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:34:50.154196 8394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:50.155221 8379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:34:50.155370 8379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "89e1f14f3b4643db99b18ae75a45a5f5"
format_stamp: "Formatted at 2026-09-29 16:34:50 on dist-test-slave-mlmz"
server_key: "01cd9cbc7fbff48bb98fcadb5456d370"
server_key_iv: "a413e2d4455e6fb24dc2ba80f08fae27"
server_key_version: "encryptionkey@0"
I20260929 16:34:50.155472 8379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:50.185814 8379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:50.188571 8379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:50.188762 8379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:50.189378 8379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:50.190155 8379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:50.190224 8379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:50.190294 8379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:50.190352 8379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:50.199339 8379 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:37839
I20260929 16:34:50.199360 8507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:37839 every 8 connection(s)
I20260929 16:34:50.200125 8379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:34:50.208397 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 8379
I20260929 16:34:50.208544 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:34:50.208837 23728 external_mini_cluster.cc:1428] Setting key 2be7b6965595dea193a5e0f17e7cf95a
Sep 29 16:34:50 dist-test-slave-mlmz krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:50.213081 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.201830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:54973 (local address 127.23.44.62:41661)
0929 16:34:50.201959 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:50.201963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:50.203110 (+ 1147us) server_negotiation.cc:407] Connection header received
0929 16:34:50.203977 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:50.203980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:50.204036 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:50.204146 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:50.205230 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.205805 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.206903 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.207139 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.207248 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.209621 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:50.209625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:50.209627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:50.209661 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:50.211122 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:50.211636 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:50.211640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:50.211641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:50.211721 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:50.212049 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:50.212054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:50.212056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:50.212223 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:50.212332 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:50.212779 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 16:34:50.212915 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:34:50.213860 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.202164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:41661 (local address 127.23.44.1:54973)
0929 16:34:50.202974 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:50.203013 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:50.203783 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:50.204324 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:50.204330 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:50.204656 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:50.205087 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.205100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.205936 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.205940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.206777 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.206786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.207282 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.207286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.207448 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.207956 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:50.207970 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:50.209463 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:50.211256 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:50.211261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:50.211274 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:50.211525 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:50.211810 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:50.211815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:50.211820 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:50.211953 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:50.212333 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:50.212341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:50.212561 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 16:34:50.212739 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 16:34:50.213037 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":509,"threads_started":1}
I20260929 16:34:50.215040 8508 heartbeater.cc:347] Connected to a master server at 127.23.44.62:41661
I20260929 16:34:50.215268 8508 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:50.215706 8508 heartbeater.cc:511] Master 127.23.44.62:41661 requested a full tablet report, sending...
I20260929 16:34:50.217324 8317 ts_manager.cc:194] Registered new tserver with Master: 89e1f14f3b4643db99b18ae75a45a5f5 (127.23.44.1:37839)
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
I20260929 16:34:50.218551 8317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:54973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:34:50.235669 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:41661
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:50.340348 8513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:50.340562 8513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:50.340602 8513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:50.344486 8513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:50.344537 8513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:50.344590 8513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:34:50.349826 8513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:41661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:50.350718 8513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:50.351591 8513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:50.360421 8521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:50.360435 8518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:50.360468 8519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:50.360740 8513 server_base.cc:1034] running on GCE node
I20260929 16:34:50.361090 8513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:50.361685 8513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:50.362843 8513 hybrid_clock.cc:648] HybridClock initialized: now 1790699690362829 us; error 28 us; skew 500 ppm
Sep 29 16:34:50 dist-test-slave-mlmz krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:50.365516 8513 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:34:50.366333 8513 webserver.cc:460] Webserver started at http://127.23.44.2:39869/ using document root <none> and password file <none>
I20260929 16:34:50.366842 8513 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:50.366899 8513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:50.367096 8513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:50.368902 8513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "f381955f89a64e7fa576bfdb01ddecf9"
format_stamp: "Formatted at 2026-09-29 16:34:50 on dist-test-slave-mlmz"
server_key: "e85684c4d23a1c49527b4a5d9b868e33"
server_key_iv: "20b59c23df5e5264752819d6ce158044"
server_key_version: "encryptionkey@0"
I20260929 16:34:50.369455 8513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "f381955f89a64e7fa576bfdb01ddecf9"
format_stamp: "Formatted at 2026-09-29 16:34:50 on dist-test-slave-mlmz"
server_key: "e85684c4d23a1c49527b4a5d9b868e33"
server_key_iv: "20b59c23df5e5264752819d6ce158044"
server_key_version: "encryptionkey@0"
I20260929 16:34:50.372776 8513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:34:50.375005 8528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:50.376101 8513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:34:50.376323 8513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "f381955f89a64e7fa576bfdb01ddecf9"
format_stamp: "Formatted at 2026-09-29 16:34:50 on dist-test-slave-mlmz"
server_key: "e85684c4d23a1c49527b4a5d9b868e33"
server_key_iv: "20b59c23df5e5264752819d6ce158044"
server_key_version: "encryptionkey@0"
I20260929 16:34:50.376425 8513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:50.394444 8513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:50.397027 8513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:50.397209 8513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:50.397792 8513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:50.398590 8513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:50.398653 8513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:50.398721 8513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:50.398767 8513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:50.407156 8513 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:35347
I20260929 16:34:50.407184 8641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:35347 every 8 connection(s)
I20260929 16:34:50.407934 8513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:34:50.411058 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 8513
I20260929 16:34:50.411195 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:34:50.411497 23728 external_mini_cluster.cc:1428] Setting key c27caeeef810366378516077b1aca419
Sep 29 16:34:50 dist-test-slave-mlmz krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:34:50.422025 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.409595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:33077 (local address 127.23.44.62:41661)
0929 16:34:50.409735 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:50.409738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:50.410480 (+ 742us) server_negotiation.cc:407] Connection header received
0929 16:34:50.411912 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:50.411916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:50.411975 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:50.412059 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:50.413439 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.414273 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.415430 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.415659 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.415780 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.418431 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:50.418434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:50.418437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:50.418458 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:50.419946 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:50.420498 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:50.420502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:50.420504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:50.420581 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:50.420908 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:50.420912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:50.420914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:50.421083 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:50.421188 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:50.421733 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 16:34:50.421859 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
I20260929 16:34:50.422857 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.409943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:41661 (local address 127.23.44.2:33077)
0929 16:34:50.410339 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:50.410375 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:50.411714 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:50.412268 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:50.412276 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:50.412741 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:50.413229 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.413241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.414412 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.414415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.415303 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.415313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.415811 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.415814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.415970 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.416530 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:50.416545 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:50.418198 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:50.420072 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:50.420077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:50.420100 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:50.420353 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:50.420671 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:50.420675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:50.420677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:50.420811 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:50.421219 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:50.421225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:50.421491 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 16:34:50.421678 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 16:34:50.421970 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
I20260929 16:34:50.423971 8642 heartbeater.cc:347] Connected to a master server at 127.23.44.62:41661
I20260929 16:34:50.424212 8642 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:50.424639 8642 heartbeater.cc:511] Master 127.23.44.62:41661 requested a full tablet report, sending...
I20260929 16:34:50.425616 8317 ts_manager.cc:194] Registered new tserver with Master: f381955f89a64e7fa576bfdb01ddecf9 (127.23.44.2:35347)
I20260929 16:34:50.426159 8317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:33077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:34:50.436776 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:41661
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:50.542119 8647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:50.542327 8647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:50.542367 8647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:50.546293 8647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:50.546339 8647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:50.546399 8647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:34:50.554154 8647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:41661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:50.555008 8647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:50.555873 8647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:50.564682 8655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:50.564693 8653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:50.564702 8652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:50.565045 8647 server_base.cc:1034] running on GCE node
I20260929 16:34:50.565465 8647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:50.566097 8647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:50.567262 8647 hybrid_clock.cc:648] HybridClock initialized: now 1790699690567221 us; error 49 us; skew 500 ppm
Sep 29 16:34:50 dist-test-slave-mlmz krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:50.569918 8647 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:34:50.570767 8647 webserver.cc:460] Webserver started at http://127.23.44.3:38665/ using document root <none> and password file <none>
I20260929 16:34:50.571230 8647 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:50.571285 8647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:50.571451 8647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:50.573378 8647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee"
format_stamp: "Formatted at 2026-09-29 16:34:50 on dist-test-slave-mlmz"
server_key: "9d56777e1e4eac8ff7b530a5fe42c2a7"
server_key_iv: "958145852127f26ca7e2d4dc54c86e1b"
server_key_version: "encryptionkey@0"
I20260929 16:34:50.573856 8647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee"
format_stamp: "Formatted at 2026-09-29 16:34:50 on dist-test-slave-mlmz"
server_key: "9d56777e1e4eac8ff7b530a5fe42c2a7"
server_key_iv: "958145852127f26ca7e2d4dc54c86e1b"
server_key_version: "encryptionkey@0"
I20260929 16:34:50.577121 8647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:50.579417 8662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:50.580579 8647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:50.580668 8647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee"
format_stamp: "Formatted at 2026-09-29 16:34:50 on dist-test-slave-mlmz"
server_key: "9d56777e1e4eac8ff7b530a5fe42c2a7"
server_key_iv: "958145852127f26ca7e2d4dc54c86e1b"
server_key_version: "encryptionkey@0"
I20260929 16:34:50.580744 8647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:50.606338 8647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:50.608976 8647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:50.609113 8647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:50.609652 8647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:50.610343 8647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:50.610388 8647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:50.610424 8647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:50.610440 8647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:50.619620 8647 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:46263
I20260929 16:34:50.619678 8775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:46263 every 8 connection(s)
I20260929 16:34:50.620790 8647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:34:50.623142 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 8647
I20260929 16:34:50.623247 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:34:50.623479 23728 external_mini_cluster.cc:1428] Setting key b77c5d54346486a5dd9f1a8fd468e88d
Sep 29 16:34:50 dist-test-slave-mlmz krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:50.634501 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.622828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:57009 (local address 127.23.44.62:41661)
0929 16:34:50.623000 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:50.623007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:50.624336 (+ 1329us) server_negotiation.cc:407] Connection header received
0929 16:34:50.625382 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:50.625385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:50.625428 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:50.625486 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:50.626477 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.626987 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.628158 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.628371 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.628468 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.630880 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:50.630883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:50.630885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:50.630910 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:50.632710 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:50.633226 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:50.633229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:50.633230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:50.633365 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:50.633634 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:50.633636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:50.633637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:50.633792 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:50.633865 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:50.634294 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 16:34:50.634387 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:34:50.635182 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.623127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:41661 (local address 127.23.44.3:57009)
0929 16:34:50.624187 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:50.624227 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:50.625128 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:50.625607 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:50.625613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:50.625933 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:50.626354 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.626366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.627114 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.627117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.627989 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.627999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.628531 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.628534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.628681 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.629319 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:50.629341 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:50.630733 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:50.632844 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:50.632851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:50.632866 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:50.633151 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:50.633439 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:50.633441 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:50.633442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:50.633562 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:50.633876 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:50.633880 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:50.634074 (+ 194us) client_negotiation.cc:769] Sending connection context
0929 16:34:50.634215 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 16:34:50.634464 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":672,"threads_started":1}
I20260929 16:34:50.636236 8776 heartbeater.cc:347] Connected to a master server at 127.23.44.62:41661
I20260929 16:34:50.636502 8776 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:50.637017 8776 heartbeater.cc:511] Master 127.23.44.62:41661 requested a full tablet report, sending...
I20260929 16:34:50.638002 8317 ts_manager.cc:194] Registered new tserver with Master: b110939bf69c4fc9bb95f8b15dd6a7ee (127.23.44.3:46263)
I20260929 16:34:50.638548 8317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:57009
I20260929 16:34:50.648856 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:34:50.657593 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.650787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.23.44.62:41661)
0929 16:34:50.650917 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:50.650920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:50.651089 (+ 169us) server_negotiation.cc:407] Connection header received
0929 16:34:50.651220 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:50.651224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:50.651260 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:50.651325 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:50.651928 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.652419 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.653204 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.653485 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.653569 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.654804 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:50.654807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:50.654809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:50.654829 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:50.656226 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:50.656585 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:50.656587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:50.656588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:50.656655 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:50.656885 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:50.656887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:50.656888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:50.657076 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:50.657181 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:50.657405 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 16:34:50.657490 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:34:50.660266 8317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:34:50.663239 8317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:34:50.677129 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.671068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.23.44.2:35347)
0929 16:34:50.672086 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:50.672104 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:50.672129 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:34:50.672251 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:50.672257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:50.672520 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:50.673002 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:50.673158 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.674361 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.675879 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.676454 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.676574 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.676710 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:50.676894 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 16:34:50.676978 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":92,"threads_started":1}
I20260929 16:34:50.677124 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.670911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:35347 (local address 127.0.0.1:57734)
0929 16:34:50.671443 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:50.671491 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:50.671601 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:50.672692 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:50.672700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:50.672725 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:50.672853 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.672860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.674496 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.674499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.675732 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.675742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.676585 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.676589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.676688 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.676741 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 16:34:50.676867 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 16:34:50.676978 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":5,"thread_start_us":75,"threads_started":1}
I20260929 16:34:50.677368 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.671563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:37839 (local address 127.0.0.1:39292)
0929 16:34:50.672476 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:50.672486 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:50.672598 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:50.673010 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:50.673012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:50.673030 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:50.673132 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.673141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.674202 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.674206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.675608 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.675617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.676815 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.676822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.676910 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.676932 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:34:50.676997 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 16:34:50.677068 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":277,"thread_start_us":90,"threads_started":1}
I20260929 16:34:50.677493 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.671987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.23.44.1:37839)
0929 16:34:50.672367 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:50.672380 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:50.672574 (+ 194us) server_negotiation.cc:407] Connection header received
0929 16:34:50.672729 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:50.672733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:50.672877 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:50.673031 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:50.673227 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.674067 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.675733 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.676499 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.676606 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.676745 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:50.677191 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 16:34:50.677320 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":94,"threads_started":1}
I20260929 16:34:50.677681 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.671100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:46263 (local address 127.0.0.1:48728)
0929 16:34:50.672182 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:50.672195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:50.672285 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:50.672904 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:50.672910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:50.672929 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:50.673009 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.673015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.674492 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.674495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.675898 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.675906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.677049 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.677052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.677145 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.677168 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:34:50.677458 (+ 290us) client_negotiation.cc:240] Negotiation successful
0929 16:34:50.677539 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":267,"thread_start_us":98,"threads_started":1}
I20260929 16:34:50.678723 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.671608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.23.44.3:46263)
0929 16:34:50.671976 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:50.671988 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:50.672378 (+ 390us) server_negotiation.cc:407] Connection header received
0929 16:34:50.672454 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:50.672458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:50.672610 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:50.672763 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:50.673128 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.674333 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.676010 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.676773 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.676872 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.676995 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:50.677848 (+ 853us) server_negotiation.cc:299] Negotiation successful
0929 16:34:50.677943 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":152,"threads_started":1}
I20260929 16:34:50.680323 8576 tablet_service.cc:1467] Processing CreateTablet for tablet ffc55ffb367b4a31a51338f25b48042d (DEFAULT_TABLE table=test-table [id=eb84e4946c9a4aa7831eb0fd695b7af1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:50.680498 8442 tablet_service.cc:1467] Processing CreateTablet for tablet ffc55ffb367b4a31a51338f25b48042d (DEFAULT_TABLE table=test-table [id=eb84e4946c9a4aa7831eb0fd695b7af1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:50.681157 8576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc55ffb367b4a31a51338f25b48042d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:50.681473 8442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc55ffb367b4a31a51338f25b48042d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:50.681104 8710 tablet_service.cc:1467] Processing CreateTablet for tablet ffc55ffb367b4a31a51338f25b48042d (DEFAULT_TABLE table=test-table [id=eb84e4946c9a4aa7831eb0fd695b7af1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:50.681988 8710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc55ffb367b4a31a51338f25b48042d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:50.687064 8793 tablet_bootstrap.cc:492] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9: Bootstrap starting.
I20260929 16:34:50.688409 8795 tablet_bootstrap.cc:492] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee: Bootstrap starting.
I20260929 16:34:50.688800 8794 tablet_bootstrap.cc:492] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5: Bootstrap starting.
I20260929 16:34:50.690460 8795 tablet_bootstrap.cc:654] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:50.690460 8793 tablet_bootstrap.cc:654] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:50.690737 8794 tablet_bootstrap.cc:654] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:50.691426 8793 log.cc:824] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:50.691617 8795 log.cc:824] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:50.691844 8794 log.cc:824] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:50.693351 8793 tablet_bootstrap.cc:492] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9: No bootstrap required, opened a new log
I20260929 16:34:50.693466 8794 tablet_bootstrap.cc:492] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5: No bootstrap required, opened a new log
I20260929 16:34:50.693539 8793 ts_tablet_manager.cc:1397] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 16:34:50.693634 8794 ts_tablet_manager.cc:1397] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:34:50.694420 8795 tablet_bootstrap.cc:492] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee: No bootstrap required, opened a new log
I20260929 16:34:50.694584 8795 ts_tablet_manager.cc:1397] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 16:34:50.696447 8793 raft_consensus.cc:348] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f381955f89a64e7fa576bfdb01ddecf9" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35347 } } peers { permanent_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 46263 } } peers { permanent_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 37839 } }
I20260929 16:34:50.696677 8793 raft_consensus.cc:374] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:50.696729 8793 raft_consensus.cc:729] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f381955f89a64e7fa576bfdb01ddecf9, State: Initialized, Role: FOLLOWER
I20260929 16:34:50.697148 8793 consensus_queue.cc:260] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f381955f89a64e7fa576bfdb01ddecf9" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35347 } } peers { permanent_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 46263 } } peers { permanent_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 37839 } }
I20260929 16:34:50.697278 8794 raft_consensus.cc:348] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f381955f89a64e7fa576bfdb01ddecf9" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35347 } } peers { permanent_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 46263 } } peers { permanent_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 37839 } }
I20260929 16:34:50.697512 8794 raft_consensus.cc:374] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:50.697570 8794 raft_consensus.cc:729] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e1f14f3b4643db99b18ae75a45a5f5, State: Initialized, Role: FOLLOWER
I20260929 16:34:50.697893 8642 heartbeater.cc:503] Master 127.23.44.62:41661 was elected leader, sending a full tablet report...
I20260929 16:34:50.697992 8794 consensus_queue.cc:260] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f381955f89a64e7fa576bfdb01ddecf9" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35347 } } peers { permanent_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 46263 } } peers { permanent_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 37839 } }
I20260929 16:34:50.698086 8793 ts_tablet_manager.cc:1428] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260929 16:34:50.698130 8795 raft_consensus.cc:348] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f381955f89a64e7fa576bfdb01ddecf9" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35347 } } peers { permanent_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 46263 } } peers { permanent_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 37839 } }
I20260929 16:34:50.698380 8795 raft_consensus.cc:374] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:50.698436 8795 raft_consensus.cc:729] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b110939bf69c4fc9bb95f8b15dd6a7ee, State: Initialized, Role: FOLLOWER
I20260929 16:34:50.698868 8794 ts_tablet_manager.cc:1428] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:34:50.698921 8795 consensus_queue.cc:260] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f381955f89a64e7fa576bfdb01ddecf9" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35347 } } peers { permanent_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 46263 } } peers { permanent_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 37839 } }
I20260929 16:34:50.699167 8508 heartbeater.cc:503] Master 127.23.44.62:41661 was elected leader, sending a full tablet report...
I20260929 16:34:50.699476 8776 heartbeater.cc:503] Master 127.23.44.62:41661 was elected leader, sending a full tablet report...
W20260929 16:34:50.701182 8509 tablet.cc:2369] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:34:50.702013 8795 ts_tablet_manager.cc:1428] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260929 16:34:50.721004 8800 raft_consensus.cc:482] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:34:50.721136 8800 raft_consensus.cc:504] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f381955f89a64e7fa576bfdb01ddecf9" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35347 } } peers { permanent_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 46263 } } peers { permanent_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 37839 } }
I20260929 16:34:50.722102 8800 leader_election.cc:288] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f381955f89a64e7fa576bfdb01ddecf9 (127.23.44.2:35347), b110939bf69c4fc9bb95f8b15dd6a7ee (127.23.44.3:46263)
I20260929 16:34:50.725914 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.722293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:35347 (local address 127.23.44.1:45251)
0929 16:34:50.722598 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:50.722609 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:50.722744 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:50.722990 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:50.722993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:50.723008 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:50.723128 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.723133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.723860 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.723863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.724738 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.724747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.725447 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.725451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.725549 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.725571 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:34:50.725661 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:34:50.725764 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":33,"thread_start_us":105,"threads_started":1}
I20260929 16:34:50.725951 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.722459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:49553 (local address 127.23.44.3:46263)
0929 16:34:50.722574 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:50.722578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:50.722740 (+ 162us) server_negotiation.cc:407] Connection header received
0929 16:34:50.722897 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:50.722905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:50.722941 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:50.723018 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:50.723324 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.723783 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.724891 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.725356 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.725469 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.725531 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:50.725780 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 16:34:50.725850 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 16:34:50.725986 8803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.722312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:46263 (local address 127.23.44.1:49553)
0929 16:34:50.722651 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:50.722662 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:50.722786 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:50.723039 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:50.723042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:50.723057 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:50.723175 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.723179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.723900 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.723903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.724768 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:50.724777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.725470 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.725474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:50.725623 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.725669 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 16:34:50.725760 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:34:50.725836 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":50,"thread_start_us":106,"threads_started":1}
I20260929 16:34:50.726476 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.722418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:45251 (local address 127.23.44.2:35347)
0929 16:34:50.722536 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:50.722541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:50.722683 (+ 142us) server_negotiation.cc:407] Connection header received
0929 16:34:50.722846 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:50.722849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:50.722890 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:50.722969 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:50.723233 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.723733 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.724861 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.725321 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.725443 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.725499 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:50.726298 (+ 799us) server_negotiation.cc:299] Negotiation successful
0929 16:34:50.726373 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:34:50.726531 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc55ffb367b4a31a51338f25b48042d" candidate_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee" is_pre_election: true
I20260929 16:34:50.726814 8730 raft_consensus.cc:2454] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e1f14f3b4643db99b18ae75a45a5f5 in term 0.
I20260929 16:34:50.727020 8596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc55ffb367b4a31a51338f25b48042d" candidate_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f381955f89a64e7fa576bfdb01ddecf9" is_pre_election: true
I20260929 16:34:50.727283 8596 raft_consensus.cc:2454] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e1f14f3b4643db99b18ae75a45a5f5 in term 0.
I20260929 16:34:50.727327 8397 leader_election.cc:302] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e1f14f3b4643db99b18ae75a45a5f5, b110939bf69c4fc9bb95f8b15dd6a7ee; no voters:
I20260929 16:34:50.727598 8800 raft_consensus.cc:2781] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:34:50.727689 8800 raft_consensus.cc:482] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:34:50.727739 8800 raft_consensus.cc:3037] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:50.728649 8800 raft_consensus.cc:504] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f381955f89a64e7fa576bfdb01ddecf9" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35347 } } peers { permanent_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 46263 } } peers { permanent_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 37839 } }
I20260929 16:34:50.728973 8800 leader_election.cc:288] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [CANDIDATE]: Term 1 election: Requested vote from peers f381955f89a64e7fa576bfdb01ddecf9 (127.23.44.2:35347), b110939bf69c4fc9bb95f8b15dd6a7ee (127.23.44.3:46263)
I20260929 16:34:50.729297 8596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc55ffb367b4a31a51338f25b48042d" candidate_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f381955f89a64e7fa576bfdb01ddecf9"
I20260929 16:34:50.729357 8730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc55ffb367b4a31a51338f25b48042d" candidate_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee"
I20260929 16:34:50.729418 8596 raft_consensus.cc:3037] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:50.729501 8730 raft_consensus.cc:3037] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:50.730403 8596 raft_consensus.cc:2454] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e1f14f3b4643db99b18ae75a45a5f5 in term 1.
I20260929 16:34:50.730507 8730 raft_consensus.cc:2454] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e1f14f3b4643db99b18ae75a45a5f5 in term 1.
I20260929 16:34:50.730698 8395 leader_election.cc:302] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e1f14f3b4643db99b18ae75a45a5f5, f381955f89a64e7fa576bfdb01ddecf9; no voters:
I20260929 16:34:50.730866 8800 raft_consensus.cc:2781] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:50.731091 8800 raft_consensus.cc:686] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [term 1 LEADER]: Becoming Leader. State: Replica: 89e1f14f3b4643db99b18ae75a45a5f5, State: Running, Role: LEADER
I20260929 16:34:50.731420 8800 consensus_queue.cc:237] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f381955f89a64e7fa576bfdb01ddecf9" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35347 } } peers { permanent_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 46263 } } peers { permanent_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 37839 } }
I20260929 16:34:50.734195 8315 catalog_manager.cc:5495] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e1f14f3b4643db99b18ae75a45a5f5 (127.23.44.1). New cstate: current_term: 1 leader_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f381955f89a64e7fa576bfdb01ddecf9" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 46263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89e1f14f3b4643db99b18ae75a45a5f5" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 37839 } health_report { overall_health: HEALTHY } } }
I20260929 16:34:50.745375 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.742443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.23.44.1:37839)
0929 16:34:50.742560 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:50.742563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:50.742574 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:34:50.742669 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:50.742671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:50.742713 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:50.742792 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:50.742974 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.743426 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.744196 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.744388 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:50.744460 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:50.744700 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:50.745034 (+ 334us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:50.745118 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:50.745183 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 16:34:50.745221 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:34:50.750487 8596 raft_consensus.cc:1261] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9 [term 1 FOLLOWER]: Refusing update from remote peer 89e1f14f3b4643db99b18ae75a45a5f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:50.750595 8730 raft_consensus.cc:1261] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee [term 1 FOLLOWER]: Refusing update from remote peer 89e1f14f3b4643db99b18ae75a45a5f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:50.751165 8804 consensus_queue.cc:1035] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b110939bf69c4fc9bb95f8b15dd6a7ee" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:50.751377 8800 consensus_queue.cc:1035] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f381955f89a64e7fa576bfdb01ddecf9" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:50.757776 8809 mvcc.cc:205] Tried to move back new op lower bound from 7334705933307932672 to 7334705933238214656. Current Snapshot: MvccSnapshot[applied={T|T < 7334705933307932672}]
I20260929 16:34:50.760358 8808 mvcc.cc:205] Tried to move back new op lower bound from 7334705933307932672 to 7334705933238214656. Current Snapshot: MvccSnapshot[applied={T|T < 7334705933307932672}]
I20260929 16:34:50.764720 8317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:50.764873 8317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:50.767125 8317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92344a3ff54e4e8ab73963750ec091fa: Sending DeleteTablet for 3 replicas of tablet ffc55ffb367b4a31a51338f25b48042d
I20260929 16:34:50.767831 8710 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc55ffb367b4a31a51338f25b48042d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:50 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:48728
I20260929 16:34:50.767844 8576 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc55ffb367b4a31a51338f25b48042d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:50 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:57734
I20260929 16:34:50.768256 8816 tablet_replica.cc:330] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9: stopping tablet replica
I20260929 16:34:50.768268 8442 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc55ffb367b4a31a51338f25b48042d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:50 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:39292
I20260929 16:34:50.768450 8816 raft_consensus.cc:2229] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:34:50.768673 8818 tablet_replica.cc:330] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5: stopping tablet replica
I20260929 16:34:50.768693 8816 raft_consensus.cc:2258] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:50.768921 8818 raft_consensus.cc:2229] T ffc55ffb367b4a31a51338f25b48042d P 89e1f14f3b4643db99b18ae75a45a5f5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:34:50.769025 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 8379
I20260929 16:34:50.770406 8816 ts_tablet_manager.cc:1905] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:50.771147 8817 tablet_replica.cc:330] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee: stopping tablet replica
I20260929 16:34:50.771390 8817 raft_consensus.cc:2229] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:34:50.771641 8817 raft_consensus.cc:2258] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:50.772687 8817 ts_tablet_manager.cc:1905] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:50.772840 8816 ts_tablet_manager.cc:1918] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:50.772936 8816 log.cc:1196] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/ffc55ffb367b4a31a51338f25b48042d
I20260929 16:34:50.773247 8816 ts_tablet_manager.cc:1939] T ffc55ffb367b4a31a51338f25b48042d P f381955f89a64e7fa576bfdb01ddecf9: Deleting consensus metadata
I20260929 16:34:50.774312 8301 catalog_manager.cc:4841] TS f381955f89a64e7fa576bfdb01ddecf9 (127.23.44.2:35347): tablet ffc55ffb367b4a31a51338f25b48042d (table test-table [id=eb84e4946c9a4aa7831eb0fd695b7af1]) successfully deleted
I20260929 16:34:50.774922 8817 ts_tablet_manager.cc:1918] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:50.775028 8817 log.cc:1196] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/ffc55ffb367b4a31a51338f25b48042d
I20260929 16:34:50.775332 8817 ts_tablet_manager.cc:1939] T ffc55ffb367b4a31a51338f25b48042d P b110939bf69c4fc9bb95f8b15dd6a7ee: Deleting consensus metadata
W20260929 16:34:50.775763 8301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:37839 (error 108)
I20260929 16:34:50.776345 8303 catalog_manager.cc:4841] TS b110939bf69c4fc9bb95f8b15dd6a7ee (127.23.44.3:46263): tablet ffc55ffb367b4a31a51338f25b48042d (table test-table [id=eb84e4946c9a4aa7831eb0fd695b7af1]) successfully deleted
I20260929 16:34:50.776700 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.776404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:37839 (local address 127.0.0.1:39300)
0929 16:34:50.776539 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:50.776609 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:37839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:34:50.776747 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 8513
W20260929 16:34:50.776856 8301 catalog_manager.cc:4584] TS 89e1f14f3b4643db99b18ae75a45a5f5 (127.23.44.1:37839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffc55ffb367b4a31a51338f25b48042d: Network error: Client connection negotiation failed: client connection to 127.23.44.1:37839: connect: Connection refused (error 111)
I20260929 16:34:50.782639 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 8647
I20260929 16:34:50.787853 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 8285
2026-09-29T16:34:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2190 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T16:34:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:50Z Disabled control of system clock
I20260929 16:34:50.818409 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:34363
--webserver_interface=127.23.44.62
--webserver_port=0
--builtin_ntp_servers=127.23.44.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:34363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:34:50.922652 8823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:50.922875 8823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:50.922910 8823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:50.926844 8823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:50.926895 8823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:50.926910 8823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:50.926926 8823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:50.926941 8823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:50.931452 8823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:34363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:34363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:50.932346 8823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:50.933657 8823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:50.941817 8829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:50.941920 8831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:50.941987 8828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:50.942260 8823 server_base.cc:1034] running on GCE node
I20260929 16:34:50.942831 8823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:50.943879 8823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:50.945050 8823 hybrid_clock.cc:648] HybridClock initialized: now 1790699690945030 us; error 30 us; skew 500 ppm
I20260929 16:34:50.947026 8823 webserver.cc:460] Webserver started at http://127.23.44.62:42153/ using document root <none> and password file <none>
I20260929 16:34:50.947604 8823 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:50.947685 8823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:50.947902 8823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:50.950173 8823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "89f406e891914e41bd141d9650fc83c5"
format_stamp: "Formatted at 2026-09-29 16:34:50 on dist-test-slave-mlmz"
server_key: "5fe6a196ee7c28f6eafe4d361187e0b8"
server_key_iv: "29b5912581fd6349ad97cacbadd62912"
server_key_version: "encryptionkey@0"
I20260929 16:34:50.950716 8823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "89f406e891914e41bd141d9650fc83c5"
format_stamp: "Formatted at 2026-09-29 16:34:50 on dist-test-slave-mlmz"
server_key: "5fe6a196ee7c28f6eafe4d361187e0b8"
server_key_iv: "29b5912581fd6349ad97cacbadd62912"
server_key_version: "encryptionkey@0"
I20260929 16:34:50.954181 8823 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:50.956430 8837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:50.957609 8823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:34:50.957726 8823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "89f406e891914e41bd141d9650fc83c5"
format_stamp: "Formatted at 2026-09-29 16:34:50 on dist-test-slave-mlmz"
server_key: "5fe6a196ee7c28f6eafe4d361187e0b8"
server_key_iv: "29b5912581fd6349ad97cacbadd62912"
server_key_version: "encryptionkey@0"
I20260929 16:34:50.957826 8823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:50.979406 8823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:50.980062 8823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:50.980252 8823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:50.986860 8823 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:34363
I20260929 16:34:50.986869 8889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:34363 every 8 connection(s)
I20260929 16:34:50.987738 8823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:50.990693 8890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:50.993589 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 8823
I20260929 16:34:50.993713 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:34:50.994021 23728 external_mini_cluster.cc:1428] Setting key 75cc8bbcc45602dcc0d4671c3badca92
I20260929 16:34:50.996367 8890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5: Bootstrap starting.
I20260929 16:34:50.998548 8890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:50.999353 8890 log.cc:824] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:51.001324 8890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5: No bootstrap required, opened a new log
I20260929 16:34:51.003290 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:50.995396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.23.44.62:34363)
0929 16:34:50.996012 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:50.996028 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:50.996064 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:34:50.996776 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:50.996794 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:50.997178 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:50.997533 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:50.997886 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:50.998974 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.000067 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.000583 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.000710 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.001565 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:51.001570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:51.001581 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:51.001963 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:51.002017 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:51.002038 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:51.002121 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:51.002306 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 16:34:51.002590 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":137,"threads_started":1}
I20260929 16:34:51.004891 8890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f406e891914e41bd141d9650fc83c5" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34363 } }
I20260929 16:34:51.005132 8890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:51.005178 8890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f406e891914e41bd141d9650fc83c5, State: Initialized, Role: FOLLOWER
I20260929 16:34:51.005630 8890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f406e891914e41bd141d9650fc83c5" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34363 } }
I20260929 16:34:51.005760 8890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:51.005807 8890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:51.005872 8890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:51.006594 8890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f406e891914e41bd141d9650fc83c5" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34363 } }
I20260929 16:34:51.006865 8890 leader_election.cc:302] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89f406e891914e41bd141d9650fc83c5; no voters:
I20260929 16:34:51.007081 8890 leader_election.cc:288] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:51.007220 8895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:51.007471 8895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [term 1 LEADER]: Becoming Leader. State: Replica: 89f406e891914e41bd141d9650fc83c5, State: Running, Role: LEADER
I20260929 16:34:51.007776 8895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f406e891914e41bd141d9650fc83c5" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34363 } }
I20260929 16:34:51.008168 8890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:51.009387 8896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89f406e891914e41bd141d9650fc83c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f406e891914e41bd141d9650fc83c5" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34363 } } }
I20260929 16:34:51.009501 8896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:51.009830 8897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89f406e891914e41bd141d9650fc83c5. Latest consensus state: current_term: 1 leader_uuid: "89f406e891914e41bd141d9650fc83c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f406e891914e41bd141d9650fc83c5" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34363 } } }
I20260929 16:34:51.009945 8897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5 [sys.catalog]: This master's current role is: LEADER
W20260929 16:34:51.012497 8910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:34:51.012583 8910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:34:51.012658 8904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:51.013386 8904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:51.018705 8904 catalog_manager.cc:1347] Generated new cluster ID: 512511c417d346789912a8374bdd760d
I20260929 16:34:51.018779 8904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:51.035326 8904 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:51.036108 8904 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:51.042241 8904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5: Generated new TSK 0
I20260929 16:34:51.042795 8904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:51.051728 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34363
--builtin_ntp_servers=127.23.44.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:34:51.174664 8914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:51.174896 8914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:51.174932 8914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:51.178858 8914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:51.178902 8914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:51.178953 8914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:34:51.183418 8914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:51.184342 8914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:51.185575 8914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:51.194499 8919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:51.194492 8922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:51.194538 8914 server_base.cc:1034] running on GCE node
W20260929 16:34:51.194490 8920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:51.195060 8914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:51.195624 8914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:51.196802 8914 hybrid_clock.cc:648] HybridClock initialized: now 1790699691196771 us; error 41 us; skew 500 ppm
I20260929 16:34:51.198712 8914 webserver.cc:460] Webserver started at http://127.23.44.1:41261/ using document root <none> and password file <none>
I20260929 16:34:51.199258 8914 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:51.199319 8914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:51.199522 8914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:51.201637 8914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "fb60561925834531b0cfaabed481f127"
format_stamp: "Formatted at 2026-09-29 16:34:51 on dist-test-slave-mlmz"
server_key: "75f5600ebd78ae02819007ff396f0ab5"
server_key_iv: "4fd849eb56b1d1f95a96ad2b096fdfbe"
server_key_version: "encryptionkey@0"
I20260929 16:34:51.202163 8914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "fb60561925834531b0cfaabed481f127"
format_stamp: "Formatted at 2026-09-29 16:34:51 on dist-test-slave-mlmz"
server_key: "75f5600ebd78ae02819007ff396f0ab5"
server_key_iv: "4fd849eb56b1d1f95a96ad2b096fdfbe"
server_key_version: "encryptionkey@0"
I20260929 16:34:51.205921 8914 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:51.208267 8928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:51.209651 8914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:34:51.209764 8914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "fb60561925834531b0cfaabed481f127"
format_stamp: "Formatted at 2026-09-29 16:34:51 on dist-test-slave-mlmz"
server_key: "75f5600ebd78ae02819007ff396f0ab5"
server_key_iv: "4fd849eb56b1d1f95a96ad2b096fdfbe"
server_key_version: "encryptionkey@0"
I20260929 16:34:51.209875 8914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:51.239403 8914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:51.240046 8914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:51.240242 8914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:51.240815 8914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:51.241736 8914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:51.241783 8914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:51.241849 8914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:51.241882 8914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:51.250547 8914 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:38735
I20260929 16:34:51.250576 9041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:38735 every 8 connection(s)
I20260929 16:34:51.251394 8914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:34:51.258507 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 8914
I20260929 16:34:51.258626 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:34:51.258965 23728 external_mini_cluster.cc:1428] Setting key 5fdf4a2497528428abba2dd51345209f
I20260929 16:34:51.260214 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.253144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:45293 (local address 127.23.44.62:34363)
0929 16:34:51.253322 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:51.253327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:51.254290 (+ 963us) server_negotiation.cc:407] Connection header received
0929 16:34:51.255586 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:51.255588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:51.255633 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:51.255706 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:51.256513 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.257048 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.258174 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.258394 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.258472 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.259704 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:51.259707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:51.259709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:51.259720 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:51.259739 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:51.259747 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:51.259848 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:51.260017 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 16:34:51.260076 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:34:51.260946 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34363
--builtin_ntp_servers=127.23.44.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:34:51.261044 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.253550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:34363 (local address 127.23.44.1:45293)
0929 16:34:51.254150 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:51.254188 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:51.255408 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:51.255872 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:51.255878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:51.255955 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:51.256398 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.256409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.257153 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.257156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.258064 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.258073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.258547 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.258554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.258757 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.259438 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:34:51.259440 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:51.259476 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:34:51.259559 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:51.259850 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:51.259862 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:51.259870 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:34:51.260005 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 16:34:51.260252 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260929 16:34:51.262193 9042 heartbeater.cc:347] Connected to a master server at 127.23.44.62:34363
I20260929 16:34:51.262449 9042 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:51.262918 9042 heartbeater.cc:511] Master 127.23.44.62:34363 requested a full tablet report, sending...
I20260929 16:34:51.264246 8854 ts_manager.cc:194] Registered new tserver with Master: fb60561925834531b0cfaabed481f127 (127.23.44.1:38735)
I20260929 16:34:51.265439 8854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.44.1:45293
W20260929 16:34:51.367599 9045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:51.367874 9045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:51.367939 9045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:51.371781 9045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:51.371855 9045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:51.371948 9045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:34:51.376441 9045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:51.377374 9045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:51.378693 9045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:51.387490 9051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:51.387494 9053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:51.387534 9050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:51.387603 9045 server_base.cc:1034] running on GCE node
I20260929 16:34:51.388123 9045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:51.388720 9045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:51.389905 9045 hybrid_clock.cc:648] HybridClock initialized: now 1790699691389873 us; error 42 us; skew 500 ppm
I20260929 16:34:51.391649 9045 webserver.cc:460] Webserver started at http://127.23.44.2:35741/ using document root <none> and password file <none>
I20260929 16:34:51.392190 9045 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:51.392278 9045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:51.392488 9045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:51.394381 9045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa"
format_stamp: "Formatted at 2026-09-29 16:34:51 on dist-test-slave-mlmz"
server_key: "b5e023ea056ef666361898ef42bd0d74"
server_key_iv: "61e3c5538c030637e61c2880303c590c"
server_key_version: "encryptionkey@0"
I20260929 16:34:51.394917 9045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa"
format_stamp: "Formatted at 2026-09-29 16:34:51 on dist-test-slave-mlmz"
server_key: "b5e023ea056ef666361898ef42bd0d74"
server_key_iv: "61e3c5538c030637e61c2880303c590c"
server_key_version: "encryptionkey@0"
I20260929 16:34:51.398478 9045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:51.400799 9059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:51.402016 9045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:51.402160 9045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa"
format_stamp: "Formatted at 2026-09-29 16:34:51 on dist-test-slave-mlmz"
server_key: "b5e023ea056ef666361898ef42bd0d74"
server_key_iv: "61e3c5538c030637e61c2880303c590c"
server_key_version: "encryptionkey@0"
I20260929 16:34:51.402269 9045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:51.423306 9045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:51.423895 9045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:51.424077 9045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:51.424610 9045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:51.425477 9045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:51.425547 9045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:51.425614 9045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:51.425665 9045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:51.433681 9045 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:43969
I20260929 16:34:51.433704 9172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:43969 every 8 connection(s)
I20260929 16:34:51.434465 9045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:34:51.437027 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 9045
I20260929 16:34:51.437140 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:34:51.437507 23728 external_mini_cluster.cc:1428] Setting key 9fca09c02f44dc4c1c32b2c56897275e
I20260929 16:34:51.439536 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34363
--builtin_ntp_servers=127.23.44.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:34:51.444307 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.436114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:53997 (local address 127.23.44.62:34363)
0929 16:34:51.436250 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:51.436255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:51.437432 (+ 1177us) server_negotiation.cc:407] Connection header received
0929 16:34:51.439043 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:51.439045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:51.439085 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:51.439147 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:51.440287 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.440943 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.442097 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.442527 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.442651 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.443713 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:51.443720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:51.443726 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:51.443743 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:51.443771 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:51.443784 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:51.443874 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:51.444007 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 16:34:51.444076 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:34:51.444940 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.436503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:34363 (local address 127.23.44.2:53997)
0929 16:34:51.437222 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:51.437295 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:51.438819 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:51.439350 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:51.439360 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:51.439464 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:51.440121 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.440142 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.441079 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.441082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.441979 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.441988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.442669 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.442672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.442803 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.443473 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:34:51.443475 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:51.443510 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:34:51.443566 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:51.443877 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:51.443882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:51.443886 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:34:51.443973 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:34:51.444161 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260929 16:34:51.446069 9173 heartbeater.cc:347] Connected to a master server at 127.23.44.62:34363
I20260929 16:34:51.446314 9173 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:51.446928 9173 heartbeater.cc:511] Master 127.23.44.62:34363 requested a full tablet report, sending...
I20260929 16:34:51.448022 8854 ts_manager.cc:194] Registered new tserver with Master: 1693b70e7ad54333bba3d0ae1e3d6aaa (127.23.44.2:43969)
I20260929 16:34:51.448611 8854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.44.2:53997
W20260929 16:34:51.548352 9176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:51.548563 9176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:51.548597 9176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:51.552452 9176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:51.552495 9176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:51.552579 9176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:34:51.557226 9176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:51.558149 9176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:51.559367 9176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:51.568552 9181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:51.568552 9184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:51.568602 9182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:51.568876 9176 server_base.cc:1034] running on GCE node
I20260929 16:34:51.569314 9176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:51.569932 9176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:51.571118 9176 hybrid_clock.cc:648] HybridClock initialized: now 1790699691571097 us; error 34 us; skew 500 ppm
I20260929 16:34:51.572777 9176 webserver.cc:460] Webserver started at http://127.23.44.3:34201/ using document root <none> and password file <none>
I20260929 16:34:51.573372 9176 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:51.573460 9176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:51.573683 9176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:51.575623 9176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "604483169d1c45368797eaced042c189"
format_stamp: "Formatted at 2026-09-29 16:34:51 on dist-test-slave-mlmz"
server_key: "c5f79b9e168ce3e01f18bea18406e900"
server_key_iv: "b1ea5ec5b6b914c67f8168b6ed70398d"
server_key_version: "encryptionkey@0"
I20260929 16:34:51.576166 9176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "604483169d1c45368797eaced042c189"
format_stamp: "Formatted at 2026-09-29 16:34:51 on dist-test-slave-mlmz"
server_key: "c5f79b9e168ce3e01f18bea18406e900"
server_key_iv: "b1ea5ec5b6b914c67f8168b6ed70398d"
server_key_version: "encryptionkey@0"
I20260929 16:34:51.579515 9176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:34:51.581723 9190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:51.582813 9176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:34:51.582942 9176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "604483169d1c45368797eaced042c189"
format_stamp: "Formatted at 2026-09-29 16:34:51 on dist-test-slave-mlmz"
server_key: "c5f79b9e168ce3e01f18bea18406e900"
server_key_iv: "b1ea5ec5b6b914c67f8168b6ed70398d"
server_key_version: "encryptionkey@0"
I20260929 16:34:51.583041 9176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:51.611878 9176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:51.612484 9176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:51.612680 9176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:51.613359 9176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:51.614265 9176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:51.614331 9176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:51.614392 9176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:51.614441 9176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:51.623013 9176 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:33827
I20260929 16:34:51.623042 9303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:33827 every 8 connection(s)
I20260929 16:34:51.623812 9176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:34:51.625645 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 9176
I20260929 16:34:51.625782 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:34:51.626063 23728 external_mini_cluster.cc:1428] Setting key efddb1b43ca6c9ca3532948bae2cc32a
I20260929 16:34:51.632380 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.625546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:38255 (local address 127.23.44.62:34363)
0929 16:34:51.625701 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:51.625706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:51.626761 (+ 1055us) server_negotiation.cc:407] Connection header received
0929 16:34:51.628265 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:51.628268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:51.628311 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:51.628374 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:51.629205 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.629769 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.630790 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.630995 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.631069 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.631971 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:51.631974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:51.631976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:51.631987 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:51.632004 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:51.632011 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:51.632068 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:51.632232 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 16:34:51.632275 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:34:51.633129 9306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.625949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:34363 (local address 127.23.44.3:38255)
0929 16:34:51.626604 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:51.626650 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:51.628075 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:51.628525 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:51.628531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:51.628606 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:51.629048 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.629060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.629877 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.629879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.630691 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.630699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.631117 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.631120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.631246 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.631747 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:34:51.631749 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:51.631783 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:34:51.631837 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:51.632082 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:51.632088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:51.632091 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 16:34:51.632209 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:34:51.632399 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":119,"threads_started":1}
I20260929 16:34:51.634132 9304 heartbeater.cc:347] Connected to a master server at 127.23.44.62:34363
I20260929 16:34:51.634331 9304 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:51.634771 9304 heartbeater.cc:511] Master 127.23.44.62:34363 requested a full tablet report, sending...
I20260929 16:34:51.635682 8854 ts_manager.cc:194] Registered new tserver with Master: 604483169d1c45368797eaced042c189 (127.23.44.3:33827)
I20260929 16:34:51.636217 8854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.44.3:38255
I20260929 16:34:51.639637 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:34:51.644579 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.641492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.23.44.62:34363)
0929 16:34:51.641598 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:51.641601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:51.641738 (+ 137us) server_negotiation.cc:407] Connection header received
0929 16:34:51.641813 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:51.641815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:51.641845 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:51.641909 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:51.642111 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.642564 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.643316 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.643509 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.643583 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.644259 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:51.644261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:51.644263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:51.644273 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:51.644289 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:51.644296 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:51.644360 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:51.644439 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 16:34:51.644480 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 16:34:51.647203 8854 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:34:51.649981 8854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:34:51.664113 9319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.658578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:33827 (local address 127.0.0.1:33068)
0929 16:34:51.659264 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:51.659278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:51.659360 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:51.660032 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:51.660035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:51.660057 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:51.660112 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.660115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.661183 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.661189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.662844 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.662855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.663600 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.663604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.663704 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.663737 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:34:51.663847 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:34:51.663966 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":68,"thread_start_us":92,"threads_started":1}
I20260929 16:34:51.664122 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.658925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.23.44.3:33827)
0929 16:34:51.659448 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:51.659459 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:51.659502 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:34:51.659572 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:51.659576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:51.659764 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:51.659899 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:51.660219 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.661077 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.662969 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.663479 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.663604 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.663781 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:51.663876 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 16:34:51.663966 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":108,"threads_started":1}
I20260929 16:34:51.664516 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.658020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43969 (local address 127.0.0.1:52940)
0929 16:34:51.658590 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:51.658605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:51.658701 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:51.659741 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:51.659746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:51.659761 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:51.659888 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.659895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.661388 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.661395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.662609 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.662619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.663979 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.663983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.664135 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.664192 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 16:34:51.664276 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 16:34:51.664364 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":117,"thread_start_us":96,"threads_started":1}
I20260929 16:34:51.665026 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.657910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.23.44.1:38735)
0929 16:34:51.658665 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:51.658679 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:51.658705 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:34:51.658807 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:51.658815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:51.659032 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:51.659197 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:51.659794 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.661170 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.662937 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.663742 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.663857 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.664021 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:51.664590 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 16:34:51.664847 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":89,"threads_started":1}
I20260929 16:34:51.664522 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.657768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:38735 (local address 127.0.0.1:35602)
0929 16:34:51.658459 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:51.658503 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:51.658611 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:51.659380 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:51.659387 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:51.659412 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:51.659617 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.659625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.661327 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.661330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.662803 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.662814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.664075 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.664078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.664268 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.664291 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:34:51.664355 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:34:51.664411 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":163,"thread_start_us":69,"threads_started":1}
I20260929 16:34:51.665243 9318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.658171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.23.44.2:43969)
0929 16:34:51.659183 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:51.659197 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:51.659220 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:34:51.659306 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:51.659312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:51.659587 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:51.659746 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:51.660001 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.661217 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.662751 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.663607 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.663722 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.663909 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:51.664957 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0929 16:34:51.665061 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":141,"threads_started":1}
I20260929 16:34:51.667510 9238 tablet_service.cc:1467] Processing CreateTablet for tablet f4d813f56c4e4ace80e3517e3142a156 (DEFAULT_TABLE table=test-table [id=9879bf7d5db3437d9f1a7754016517e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:51.667660 9107 tablet_service.cc:1467] Processing CreateTablet for tablet f4d813f56c4e4ace80e3517e3142a156 (DEFAULT_TABLE table=test-table [id=9879bf7d5db3437d9f1a7754016517e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:51.667876 8976 tablet_service.cc:1467] Processing CreateTablet for tablet f4d813f56c4e4ace80e3517e3142a156 (DEFAULT_TABLE table=test-table [id=9879bf7d5db3437d9f1a7754016517e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:51.668561 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d813f56c4e4ace80e3517e3142a156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:51.668561 9107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d813f56c4e4ace80e3517e3142a156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:51.668907 8976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d813f56c4e4ace80e3517e3142a156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:51.675026 9321 tablet_bootstrap.cc:492] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127: Bootstrap starting.
I20260929 16:34:51.676151 9322 tablet_bootstrap.cc:492] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa: Bootstrap starting.
I20260929 16:34:51.677460 9321 tablet_bootstrap.cc:654] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:51.677605 9323 tablet_bootstrap.cc:492] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189: Bootstrap starting.
I20260929 16:34:51.678308 9322 tablet_bootstrap.cc:654] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:51.678365 9321 log.cc:824] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:51.679175 9322 log.cc:824] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:51.679600 9323 tablet_bootstrap.cc:654] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:51.680338 9323 log.cc:824] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:51.680516 9321 tablet_bootstrap.cc:492] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127: No bootstrap required, opened a new log
I20260929 16:34:51.680707 9321 ts_tablet_manager.cc:1397] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 16:34:51.681083 9322 tablet_bootstrap.cc:492] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa: No bootstrap required, opened a new log
I20260929 16:34:51.681295 9322 ts_tablet_manager.cc:1397] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:34:51.682626 9323 tablet_bootstrap.cc:492] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189: No bootstrap required, opened a new log
I20260929 16:34:51.682860 9323 ts_tablet_manager.cc:1397] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:34:51.686038 9321 raft_consensus.cc:348] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb60561925834531b0cfaabed481f127" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38735 } } peers { permanent_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43969 } } peers { permanent_uuid: "604483169d1c45368797eaced042c189" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 33827 } }
I20260929 16:34:51.686401 9321 raft_consensus.cc:374] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:51.686467 9321 raft_consensus.cc:729] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb60561925834531b0cfaabed481f127, State: Initialized, Role: FOLLOWER
I20260929 16:34:51.686748 9322 raft_consensus.cc:348] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb60561925834531b0cfaabed481f127" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38735 } } peers { permanent_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43969 } } peers { permanent_uuid: "604483169d1c45368797eaced042c189" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 33827 } }
I20260929 16:34:51.687028 9322 raft_consensus.cc:374] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:51.686987 9321 consensus_queue.cc:260] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb60561925834531b0cfaabed481f127" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38735 } } peers { permanent_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43969 } } peers { permanent_uuid: "604483169d1c45368797eaced042c189" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 33827 } }
I20260929 16:34:51.687103 9322 raft_consensus.cc:729] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1693b70e7ad54333bba3d0ae1e3d6aaa, State: Initialized, Role: FOLLOWER
I20260929 16:34:51.687605 9322 consensus_queue.cc:260] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb60561925834531b0cfaabed481f127" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38735 } } peers { permanent_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43969 } } peers { permanent_uuid: "604483169d1c45368797eaced042c189" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 33827 } }
I20260929 16:34:51.687700 9042 heartbeater.cc:503] Master 127.23.44.62:34363 was elected leader, sending a full tablet report...
I20260929 16:34:51.687697 9323 raft_consensus.cc:348] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb60561925834531b0cfaabed481f127" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38735 } } peers { permanent_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43969 } } peers { permanent_uuid: "604483169d1c45368797eaced042c189" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 33827 } }
I20260929 16:34:51.688001 9321 ts_tablet_manager.cc:1428] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 16:34:51.688099 9323 raft_consensus.cc:374] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:51.688200 9323 raft_consensus.cc:729] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604483169d1c45368797eaced042c189, State: Initialized, Role: FOLLOWER
I20260929 16:34:51.688596 9322 ts_tablet_manager.cc:1428] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:34:51.688652 9323 consensus_queue.cc:260] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb60561925834531b0cfaabed481f127" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38735 } } peers { permanent_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43969 } } peers { permanent_uuid: "604483169d1c45368797eaced042c189" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 33827 } }
I20260929 16:34:51.688946 9173 heartbeater.cc:503] Master 127.23.44.62:34363 was elected leader, sending a full tablet report...
I20260929 16:34:51.689352 9304 heartbeater.cc:503] Master 127.23.44.62:34363 was elected leader, sending a full tablet report...
I20260929 16:34:51.689607 9323 ts_tablet_manager.cc:1428] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 16:34:51.752584 9043 tablet.cc:2369] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:34:51.874722 9305 tablet.cc:2369] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:34:51.935504 9174 tablet.cc:2369] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:34:51.974695 9329 raft_consensus.cc:482] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:34:51.974937 9329 raft_consensus.cc:504] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb60561925834531b0cfaabed481f127" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38735 } } peers { permanent_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43969 } } peers { permanent_uuid: "604483169d1c45368797eaced042c189" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 33827 } }
I20260929 16:34:51.976109 9329 leader_election.cc:288] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb60561925834531b0cfaabed481f127 (127.23.44.1:38735), 1693b70e7ad54333bba3d0ae1e3d6aaa (127.23.44.2:43969)
I20260929 16:34:51.980518 9306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.976295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:38735 (local address 127.23.44.3:47955)
0929 16:34:51.976435 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:51.976451 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:51.976555 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:51.976881 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:51.976886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:51.976912 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:51.977067 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.977077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.978064 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.978066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.979273 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.979284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.980096 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.980098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.980171 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.980221 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 16:34:51.980298 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 16:34:51.980371 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"spinlock_wait_cycles":4224}
I20260929 16:34:51.980522 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.976481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:47955 (local address 127.23.44.1:38735)
0929 16:34:51.976628 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:51.976632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:51.976648 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:34:51.976716 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:51.976720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:51.976774 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:51.976840 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:51.977190 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.977959 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.979397 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.979988 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.980093 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.980151 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:51.980331 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 16:34:51.980403 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:34:51.980685 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.976475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:43969 (local address 127.23.44.3:53125)
0929 16:34:51.976856 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:51.976869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:51.976974 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:51.977230 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:51.977232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:51.977247 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:51.977397 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.977401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.978279 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.978282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.979436 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:51.979443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.980264 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.980268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:51.980400 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.980420 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 16:34:51.980488 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 16:34:51.980548 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"spinlock_wait_cycles":23808,"thread_start_us":118,"threads_started":1}
I20260929 16:34:51.980717 9318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:51.976606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:53125 (local address 127.23.44.2:43969)
0929 16:34:51.976721 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:51.976727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:51.976963 (+ 236us) server_negotiation.cc:407] Connection header received
0929 16:34:51.977067 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:51.977069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:51.977109 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:51.977218 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:51.977513 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.978171 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.979550 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:51.980135 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:51.980240 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:51.980315 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:51.980521 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 16:34:51.980598 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 16:34:51.981182 8996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d813f56c4e4ace80e3517e3142a156" candidate_uuid: "604483169d1c45368797eaced042c189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb60561925834531b0cfaabed481f127" is_pre_election: true
I20260929 16:34:51.981333 9127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d813f56c4e4ace80e3517e3142a156" candidate_uuid: "604483169d1c45368797eaced042c189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa" is_pre_election: true
I20260929 16:34:51.981510 8996 raft_consensus.cc:2454] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 604483169d1c45368797eaced042c189 in term 0.
I20260929 16:34:51.981585 9127 raft_consensus.cc:2454] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 604483169d1c45368797eaced042c189 in term 0.
I20260929 16:34:51.982023 9194 leader_election.cc:302] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1693b70e7ad54333bba3d0ae1e3d6aaa, 604483169d1c45368797eaced042c189, fb60561925834531b0cfaabed481f127; no voters:
I20260929 16:34:51.982303 9329 raft_consensus.cc:2781] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:34:51.982400 9329 raft_consensus.cc:482] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:34:51.982460 9329 raft_consensus.cc:3037] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:51.983551 9329 raft_consensus.cc:504] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb60561925834531b0cfaabed481f127" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38735 } } peers { permanent_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43969 } } peers { permanent_uuid: "604483169d1c45368797eaced042c189" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 33827 } }
I20260929 16:34:51.983886 9329 leader_election.cc:288] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [CANDIDATE]: Term 1 election: Requested vote from peers fb60561925834531b0cfaabed481f127 (127.23.44.1:38735), 1693b70e7ad54333bba3d0ae1e3d6aaa (127.23.44.2:43969)
I20260929 16:34:51.984236 9127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d813f56c4e4ace80e3517e3142a156" candidate_uuid: "604483169d1c45368797eaced042c189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa"
I20260929 16:34:51.984275 8996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d813f56c4e4ace80e3517e3142a156" candidate_uuid: "604483169d1c45368797eaced042c189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb60561925834531b0cfaabed481f127"
I20260929 16:34:51.984346 9127 raft_consensus.cc:3037] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:51.984411 8996 raft_consensus.cc:3037] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:51.985416 9127 raft_consensus.cc:2454] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 604483169d1c45368797eaced042c189 in term 1.
I20260929 16:34:51.985417 8996 raft_consensus.cc:2454] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 604483169d1c45368797eaced042c189 in term 1.
I20260929 16:34:51.985744 9194 leader_election.cc:302] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1693b70e7ad54333bba3d0ae1e3d6aaa, 604483169d1c45368797eaced042c189; no voters:
I20260929 16:34:51.985915 9329 raft_consensus.cc:2781] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:51.986152 9329 raft_consensus.cc:686] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 1 LEADER]: Becoming Leader. State: Replica: 604483169d1c45368797eaced042c189, State: Running, Role: LEADER
I20260929 16:34:51.986457 9329 consensus_queue.cc:237] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb60561925834531b0cfaabed481f127" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38735 } } peers { permanent_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43969 } } peers { permanent_uuid: "604483169d1c45368797eaced042c189" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 33827 } }
I20260929 16:34:51.989683 8852 catalog_manager.cc:5495] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 reported cstate change: term changed from 0 to 1, leader changed from <none> to 604483169d1c45368797eaced042c189 (127.23.44.3). New cstate: current_term: 1 leader_uuid: "604483169d1c45368797eaced042c189" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb60561925834531b0cfaabed481f127" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "604483169d1c45368797eaced042c189" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 33827 } health_report { overall_health: HEALTHY } } }
I20260929 16:34:52.022119 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:52.019028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.23.44.3:33827)
0929 16:34:52.019171 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:52.019174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:52.019187 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:34:52.019258 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:52.019260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:52.019307 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:52.019400 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:52.019619 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:52.020100 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:52.020922 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:52.021136 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:52.021214 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:52.021477 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:52.021840 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:52.021926 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:52.021983 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 16:34:52.022023 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:34:52.027510 9127 raft_consensus.cc:1261] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa [term 1 FOLLOWER]: Refusing update from remote peer 604483169d1c45368797eaced042c189: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:52.027527 8996 raft_consensus.cc:1261] T f4d813f56c4e4ace80e3517e3142a156 P fb60561925834531b0cfaabed481f127 [term 1 FOLLOWER]: Refusing update from remote peer 604483169d1c45368797eaced042c189: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:52.028214 9329 consensus_queue.cc:1035] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1693b70e7ad54333bba3d0ae1e3d6aaa" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 43969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:52.028416 9331 consensus_queue.cc:1035] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb60561925834531b0cfaabed481f127" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 38735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:52.035372 9336 mvcc.cc:205] Tried to move back new op lower bound from 7334705938537684992 to 7334705938379292672. Current Snapshot: MvccSnapshot[applied={T|T < 7334705938537684992}]
I20260929 16:34:52.042529 8853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:52.042703 8853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:52.045113 8853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89f406e891914e41bd141d9650fc83c5: Sending DeleteTablet for 3 replicas of tablet f4d813f56c4e4ace80e3517e3142a156
I20260929 16:34:52.045866 9107 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d813f56c4e4ace80e3517e3142a156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:52 UTC) from {username='slave'} at 127.0.0.1:52940
I20260929 16:34:52.045914 8976 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d813f56c4e4ace80e3517e3142a156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:52 UTC) from {username='slave'} at 127.0.0.1:35602
I20260929 16:34:52.046317 9344 tablet_replica.cc:330] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa: stopping tablet replica
I20260929 16:34:52.046389 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 8914
I20260929 16:34:52.046530 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d813f56c4e4ace80e3517e3142a156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:52 UTC) from {username='slave'} at 127.0.0.1:33068
I20260929 16:34:52.047019 9344 raft_consensus.cc:2229] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:34:52.047312 9344 raft_consensus.cc:2258] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:52.047904 9346 tablet_replica.cc:330] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189: stopping tablet replica
I20260929 16:34:52.048193 9346 raft_consensus.cc:2229] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:34:52.048532 9346 raft_consensus.cc:2258] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:52.048594 9344 ts_tablet_manager.cc:1905] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:52.049942 9346 ts_tablet_manager.cc:1905] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:52.051064 9344 ts_tablet_manager.cc:1918] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:52.051165 9344 log.cc:1196] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/f4d813f56c4e4ace80e3517e3142a156
I20260929 16:34:52.051469 9344 ts_tablet_manager.cc:1939] T f4d813f56c4e4ace80e3517e3142a156 P 1693b70e7ad54333bba3d0ae1e3d6aaa: Deleting consensus metadata
I20260929 16:34:52.052450 8841 catalog_manager.cc:4841] TS 1693b70e7ad54333bba3d0ae1e3d6aaa (127.23.44.2:43969): tablet f4d813f56c4e4ace80e3517e3142a156 (table test-table [id=9879bf7d5db3437d9f1a7754016517e0]) successfully deleted
I20260929 16:34:52.054495 9346 ts_tablet_manager.cc:1918] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:52.054603 9346 log.cc:1196] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/f4d813f56c4e4ace80e3517e3142a156
I20260929 16:34:52.054875 9346 ts_tablet_manager.cc:1939] T f4d813f56c4e4ace80e3517e3142a156 P 604483169d1c45368797eaced042c189: Deleting consensus metadata
I20260929 16:34:52.055750 8838 catalog_manager.cc:4841] TS 604483169d1c45368797eaced042c189 (127.23.44.3:33827): tablet f4d813f56c4e4ace80e3517e3142a156 (table test-table [id=9879bf7d5db3437d9f1a7754016517e0]) successfully deleted
W20260929 16:34:52.056389 8838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:38735 (error 108)
I20260929 16:34:52.056572 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 9045
I20260929 16:34:52.061283 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:52.060921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:38735 (local address 127.0.0.1:35604)
0929 16:34:52.061080 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:52.061160 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:38735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:34:52.061462 8838 catalog_manager.cc:4584] TS fb60561925834531b0cfaabed481f127 (127.23.44.1:38735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4d813f56c4e4ace80e3517e3142a156: Network error: Client connection negotiation failed: client connection to 127.23.44.1:38735: connect: Connection refused (error 111)
I20260929 16:34:52.063054 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 9176
I20260929 16:34:52.068513 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 8823
2026-09-29T16:34:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1278 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:52 dist-test-slave-mlmz krb5kdc[9350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:52 dist-test-slave-mlmz krb5kdc[9350](info): set up 2 sockets
Sep 29 16:34:52 dist-test-slave-mlmz krb5kdc[9350](info): commencing operation
krb5kdc: starting...
W20260929 16:34:53.105504 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:34:53 dist-test-slave-mlmz krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:34:53.223912 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:33455
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:53.327533 9363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:53.327824 9363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:53.327903 9363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:53.331818 9363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:53.331902 9363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:53.331941 9363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:53.331998 9363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:53.332037 9363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:53.336997 9363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:33455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:33455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:53.337930 9363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:53.338853 9363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:53.346793 9371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:53.346873 9368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:53.346938 9369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:53.347031 9363 server_base.cc:1034] running on GCE node
I20260929 16:34:53.347609 9363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:53.348630 9363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:53.349819 9363 hybrid_clock.cc:648] HybridClock initialized: now 1790699693349812 us; error 36 us; skew 500 ppm
Sep 29 16:34:53 dist-test-slave-mlmz krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699693, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:53.352401 9363 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:34:53.353312 9363 webserver.cc:460] Webserver started at http://127.23.44.62:44729/ using document root <none> and password file <none>
I20260929 16:34:53.353838 9363 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:53.353940 9363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:53.354141 9363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:53.356012 9363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "42d9ff57002b469cbf0a5f479aa79217"
format_stamp: "Formatted at 2026-09-29 16:34:53 on dist-test-slave-mlmz"
server_key: "aea111b97d69e5158e4e16732b68f165"
server_key_iv: "40a144bda7d68751e5c6bc351aecfcd8"
server_key_version: "encryptionkey@0"
I20260929 16:34:53.356551 9363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "42d9ff57002b469cbf0a5f479aa79217"
format_stamp: "Formatted at 2026-09-29 16:34:53 on dist-test-slave-mlmz"
server_key: "aea111b97d69e5158e4e16732b68f165"
server_key_iv: "40a144bda7d68751e5c6bc351aecfcd8"
server_key_version: "encryptionkey@0"
I20260929 16:34:53.360016 9363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 16:34:53.362337 9378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:53.363530 9363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:34:53.363665 9363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "42d9ff57002b469cbf0a5f479aa79217"
format_stamp: "Formatted at 2026-09-29 16:34:53 on dist-test-slave-mlmz"
server_key: "aea111b97d69e5158e4e16732b68f165"
server_key_iv: "40a144bda7d68751e5c6bc351aecfcd8"
server_key_version: "encryptionkey@0"
I20260929 16:34:53.363775 9363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:53.380131 9363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:53.383721 9363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:53.383941 9363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:53.391098 9363 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:33455
I20260929 16:34:53.391116 9430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:33455 every 8 connection(s)
I20260929 16:34:53.392030 9363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:53.395030 9431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:53.399320 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 9363
I20260929 16:34:53.399467 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:34:53.399775 23728 external_mini_cluster.cc:1428] Setting key 848b3b935743cf3fa4643c590142db4f
I20260929 16:34:53.401224 9431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217: Bootstrap starting.
I20260929 16:34:53.403364 9431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:53.404079 9431 log.cc:824] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:53.406001 9431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217: No bootstrap required, opened a new log
Sep 29 16:34:53 dist-test-slave-mlmz krb5kdc[9350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:53.409693 9431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d9ff57002b469cbf0a5f479aa79217" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 33455 } }
I20260929 16:34:53.409987 9431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:53.410075 9431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d9ff57002b469cbf0a5f479aa79217, State: Initialized, Role: FOLLOWER
I20260929 16:34:53.410552 9431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d9ff57002b469cbf0a5f479aa79217" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 33455 } }
I20260929 16:34:53.410673 9431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:53.410758 9431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:53.410851 9431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:53.411661 9431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d9ff57002b469cbf0a5f479aa79217" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 33455 } }
I20260929 16:34:53.411988 9431 leader_election.cc:302] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42d9ff57002b469cbf0a5f479aa79217; no voters:
I20260929 16:34:53.412288 9431 leader_election.cc:288] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:53.412432 9436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:53.412657 9436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [term 1 LEADER]: Becoming Leader. State: Replica: 42d9ff57002b469cbf0a5f479aa79217, State: Running, Role: LEADER
I20260929 16:34:53.412949 9436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d9ff57002b469cbf0a5f479aa79217" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 33455 } }
I20260929 16:34:53.413563 9431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:53.414261 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:53.401149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.23.44.62:33455)
0929 16:34:53.401582 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:53.401600 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:53.401633 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:34:53.402247 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:53.402262 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:53.402569 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:53.402918 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:53.403732 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:53.404855 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:53.405816 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:53.406326 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:53.406490 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:53.408947 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:53.408952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:53.408964 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:53.408990 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:53.410915 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:53.411322 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:53.411330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:53.411338 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:53.411451 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:53.411742 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:53.411748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:53.411750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:53.412201 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:53.412410 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:53.413030 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 16:34:53.413340 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":117,"threads_started":1}
I20260929 16:34:53.414511 9437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42d9ff57002b469cbf0a5f479aa79217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d9ff57002b469cbf0a5f479aa79217" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 33455 } } }
I20260929 16:34:53.414629 9437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:53.414969 9438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42d9ff57002b469cbf0a5f479aa79217. Latest consensus state: current_term: 1 leader_uuid: "42d9ff57002b469cbf0a5f479aa79217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d9ff57002b469cbf0a5f479aa79217" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 33455 } } }
I20260929 16:34:53.415066 9438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:53.416309 9445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:53.419188 9445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:53.425179 9445 catalog_manager.cc:1347] Generated new cluster ID: 02af846b39ae4a55b8eb29b7bf29ffd0
I20260929 16:34:53.425264 9445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:53.435307 9445 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:53.436147 9445 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:53.447472 9445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217: Generated new TSK 0
I20260929 16:34:53.448165 9445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:34:53.479840 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:33455
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:53.598428 9457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:53.598659 9457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:53.598701 9457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:53.602510 9457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:53.602556 9457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:53.602610 9457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:34:53.607621 9457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:33455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:53.608474 9457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:53.609369 9457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:53.618575 9465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:53.618597 9462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:53.618563 9463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:53.618959 9457 server_base.cc:1034] running on GCE node
I20260929 16:34:53.619333 9457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:53.619901 9457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:53.621104 9457 hybrid_clock.cc:648] HybridClock initialized: now 1790699693621089 us; error 26 us; skew 500 ppm
Sep 29 16:34:53 dist-test-slave-mlmz krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699693, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:53.623736 9457 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:34:53.624480 9457 webserver.cc:460] Webserver started at http://127.23.44.1:41117/ using document root <none> and password file <none>
I20260929 16:34:53.624948 9457 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:53.625001 9457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:53.625149 9457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:53.627064 9457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "4d19460daa874a409e4e98a818371afa"
format_stamp: "Formatted at 2026-09-29 16:34:53 on dist-test-slave-mlmz"
server_key: "a9132f64ebafe7c2cf73aacf627f46ac"
server_key_iv: "a037dbb47b917e6143e0a6369663a538"
server_key_version: "encryptionkey@0"
I20260929 16:34:53.627549 9457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "4d19460daa874a409e4e98a818371afa"
format_stamp: "Formatted at 2026-09-29 16:34:53 on dist-test-slave-mlmz"
server_key: "a9132f64ebafe7c2cf73aacf627f46ac"
server_key_iv: "a037dbb47b917e6143e0a6369663a538"
server_key_version: "encryptionkey@0"
I20260929 16:34:53.631386 9457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:34:53.633682 9472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:53.634851 9457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:34:53.634945 9457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "4d19460daa874a409e4e98a818371afa"
format_stamp: "Formatted at 2026-09-29 16:34:53 on dist-test-slave-mlmz"
server_key: "a9132f64ebafe7c2cf73aacf627f46ac"
server_key_iv: "a037dbb47b917e6143e0a6369663a538"
server_key_version: "encryptionkey@0"
I20260929 16:34:53.635030 9457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:53.644381 9457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:53.647522 9457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:53.647732 9457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:53.648303 9457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:53.649127 9457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:53.649215 9457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:53.649348 9457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:53.649398 9457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:53.658037 9457 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:41767
I20260929 16:34:53.658098 9585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:41767 every 8 connection(s)
I20260929 16:34:53.658954 9457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:34:53.666229 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 9457
I20260929 16:34:53.666409 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:34:53.666764 23728 external_mini_cluster.cc:1428] Setting key 8339054ec185cde8e55980e548556c86
Sep 29 16:34:53 dist-test-slave-mlmz krb5kdc[9350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699693, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:53.673228 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:53.661040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:35235 (local address 127.23.44.62:33455)
0929 16:34:53.661182 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:53.661187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:53.662121 (+ 934us) server_negotiation.cc:407] Connection header received
0929 16:34:53.663374 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:53.663377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:53.663425 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:53.663502 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:53.664547 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:53.665127 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:53.666330 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:53.666724 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:53.666869 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:53.669532 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:53.669538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:53.669544 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:53.669576 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:53.671053 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:53.671651 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:53.671653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:53.671655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:53.671738 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:53.672157 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:53.672160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:53.672161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:53.672326 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:53.672401 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:53.673012 (+ 611us) server_negotiation.cc:299] Negotiation successful
0929 16:34:53.673112 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 16:34:53.674387 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:53.661512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:33455 (local address 127.23.44.1:35235)
0929 16:34:53.661975 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:53.662013 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:53.663177 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:53.663618 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:53.663623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:53.663943 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:53.664422 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:53.664435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:53.665319 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:53.665322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:53.666202 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:53.666211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:53.666914 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:53.666918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:53.667058 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:53.667750 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:53.667768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:53.669378 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:53.671167 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:53.671176 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:53.671191 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:53.671541 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:53.671860 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:53.671866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:53.671870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:53.672054 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:53.672433 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:53.672442 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:53.672746 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 16:34:53.672970 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 16:34:53.673343 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":119,"threads_started":1}
I20260929 16:34:53.675588 9586 heartbeater.cc:347] Connected to a master server at 127.23.44.62:33455
I20260929 16:34:53.675853 9586 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
I20260929 16:34:53.676373 9586 heartbeater.cc:511] Master 127.23.44.62:33455 requested a full tablet report, sending...
I20260929 16:34:53.677860 9395 ts_manager.cc:194] Registered new tserver with Master: 4d19460daa874a409e4e98a818371afa (127.23.44.1:41767)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
I20260929 16:34:53.679356 9395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:35235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:34:53.692373 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:33455
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:53.797225 9591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:53.797506 9591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:53.797580 9591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:53.801476 9591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:53.801555 9591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:53.801640 9591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:34:53.806756 9591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:33455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:53.808013 9591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:53.808971 9591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:34:53.818127 9591 server_base.cc:1034] running on GCE node
W20260929 16:34:53.818100 9599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:53.818095 9597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:53.818089 9596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:53.818643 9591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:53.819226 9591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:53.820408 9591 hybrid_clock.cc:648] HybridClock initialized: now 1790699693820390 us; error 28 us; skew 500 ppm
Sep 29 16:34:53 dist-test-slave-mlmz krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699693, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:53.822980 9591 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:34:53.823791 9591 webserver.cc:460] Webserver started at http://127.23.44.2:35841/ using document root <none> and password file <none>
I20260929 16:34:53.824353 9591 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:53.824440 9591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:53.824653 9591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:53.826567 9591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "c73b038d6004415cafd93c7471dcbcb0"
format_stamp: "Formatted at 2026-09-29 16:34:53 on dist-test-slave-mlmz"
server_key: "05535bb6d53b8649e92d35f69eb70f72"
server_key_iv: "1771ac69ea5411e17610ddc382d9a256"
server_key_version: "encryptionkey@0"
I20260929 16:34:53.827114 9591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "c73b038d6004415cafd93c7471dcbcb0"
format_stamp: "Formatted at 2026-09-29 16:34:53 on dist-test-slave-mlmz"
server_key: "05535bb6d53b8649e92d35f69eb70f72"
server_key_iv: "1771ac69ea5411e17610ddc382d9a256"
server_key_version: "encryptionkey@0"
I20260929 16:34:53.830466 9591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:53.832633 9606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:53.833751 9591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:34:53.833884 9591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "c73b038d6004415cafd93c7471dcbcb0"
format_stamp: "Formatted at 2026-09-29 16:34:53 on dist-test-slave-mlmz"
server_key: "05535bb6d53b8649e92d35f69eb70f72"
server_key_iv: "1771ac69ea5411e17610ddc382d9a256"
server_key_version: "encryptionkey@0"
I20260929 16:34:53.833985 9591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:53.855032 9591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:53.857832 9591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:53.858029 9591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:53.858579 9591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:53.859329 9591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:53.859397 9591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:53.859465 9591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:53.859516 9591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:53.868036 9591 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:37715
I20260929 16:34:53.868104 9719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:37715 every 8 connection(s)
I20260929 16:34:53.868868 9591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
Sep 29 16:34:53 dist-test-slave-mlmz krb5kdc[9350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699693, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:53.878016 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 9591
I20260929 16:34:53.878165 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:34:53.878481 23728 external_mini_cluster.cc:1428] Setting key 2f79719cff11ac63c3071fdcb49d2558
I20260929 16:34:53.882795 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:53.870538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:35225 (local address 127.23.44.62:33455)
0929 16:34:53.870668 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:53.870671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:53.871369 (+ 698us) server_negotiation.cc:407] Connection header received
0929 16:34:53.872575 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:53.872578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:53.872623 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:53.872696 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:53.873831 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:53.874391 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:53.875453 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:53.875666 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:53.875751 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:53.878411 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:53.878415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:53.878418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:53.878541 (+ 123us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:53.880143 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:53.880765 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:53.880768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:53.880770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:53.880861 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:53.881250 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:53.881253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:53.881254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:53.881537 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:53.881846 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:53.882345 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 16:34:53.882604 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:34:53.883291 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:53.870818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:33455 (local address 127.23.44.2:35225)
0929 16:34:53.871229 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:53.871266 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:53.872391 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:53.872843 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:53.872849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:53.873184 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:53.873675 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:53.873688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:53.874503 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:53.874506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:53.875342 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:53.875350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:53.875791 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:53.875794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:53.875933 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:53.876516 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:53.876531 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:53.878150 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:53.880259 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:53.880265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:53.880280 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:53.880638 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:53.880959 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:53.880965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:53.880970 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:53.881156 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:53.881809 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:53.881814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:53.882053 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 16:34:53.882214 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:34:53.882458 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20260929 16:34:53.884387 9720 heartbeater.cc:347] Connected to a master server at 127.23.44.62:33455
I20260929 16:34:53.884648 9720 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:53.885114 9720 heartbeater.cc:511] Master 127.23.44.62:33455 requested a full tablet report, sending...
I20260929 16:34:53.886222 9395 ts_manager.cc:194] Registered new tserver with Master: c73b038d6004415cafd93c7471dcbcb0 (127.23.44.2:37715)
I20260929 16:34:53.886777 9395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:35225
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:34:53.908012 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:33455
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:54.013382 9725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:54.013644 9725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:54.013715 9725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:54.018018 9725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:54.018067 9725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:54.018149 9725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:34:54.023159 9725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:33455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:54.024050 9725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:54.024987 9725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:54.033824 9730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:54.033794 9733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:54.034204 9731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:54.034248 9725 server_base.cc:1034] running on GCE node
I20260929 16:34:54.034631 9725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:54.035336 9725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:54.036547 9725 hybrid_clock.cc:648] HybridClock initialized: now 1790699694036532 us; error 36 us; skew 500 ppm
Sep 29 16:34:54 dist-test-slave-mlmz krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699694, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:54.039371 9725 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:34:54.040175 9725 webserver.cc:460] Webserver started at http://127.23.44.3:42413/ using document root <none> and password file <none>
I20260929 16:34:54.040685 9725 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:54.040783 9725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:54.041002 9725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:54.043282 9725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "b903b3379ae6466a9d7c0bf85d000d17"
format_stamp: "Formatted at 2026-09-29 16:34:54 on dist-test-slave-mlmz"
server_key: "0e9cbd70ac931be3fa8c82eeedf7a5dc"
server_key_iv: "0bf727d0bc5645698ddbc1324a42a358"
server_key_version: "encryptionkey@0"
I20260929 16:34:54.043839 9725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "b903b3379ae6466a9d7c0bf85d000d17"
format_stamp: "Formatted at 2026-09-29 16:34:54 on dist-test-slave-mlmz"
server_key: "0e9cbd70ac931be3fa8c82eeedf7a5dc"
server_key_iv: "0bf727d0bc5645698ddbc1324a42a358"
server_key_version: "encryptionkey@0"
I20260929 16:34:54.047405 9725 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:34:54.049711 9740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:54.050762 9725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:34:54.050894 9725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "b903b3379ae6466a9d7c0bf85d000d17"
format_stamp: "Formatted at 2026-09-29 16:34:54 on dist-test-slave-mlmz"
server_key: "0e9cbd70ac931be3fa8c82eeedf7a5dc"
server_key_iv: "0bf727d0bc5645698ddbc1324a42a358"
server_key_version: "encryptionkey@0"
I20260929 16:34:54.050997 9725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:54.060479 9725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:54.063488 9725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:54.063678 9725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:54.064206 9725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:54.064941 9725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:54.065011 9725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:54.065079 9725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:54.065122 9725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:54.073895 9725 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:39067
I20260929 16:34:54.073928 9853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:39067 every 8 connection(s)
I20260929 16:34:54.074687 9725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
Sep 29 16:34:54 dist-test-slave-mlmz krb5kdc[9350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699694, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:54.083638 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 9725
I20260929 16:34:54.083776 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:34:54.084076 23728 external_mini_cluster.cc:1428] Setting key 24b6975a86b931c9d0a6a8c4c7dd8ff6
I20260929 16:34:54.087538 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.076327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:39465 (local address 127.23.44.62:33455)
0929 16:34:54.076488 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:54.076492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:54.077241 (+ 749us) server_negotiation.cc:407] Connection header received
0929 16:34:54.078553 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:54.078556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:54.078599 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:54.078670 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:54.079632 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.080130 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.081181 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.081454 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.081534 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.084157 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:54.084162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:54.084166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:54.084190 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:54.085782 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:54.086262 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:54.086265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:54.086267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:54.086356 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:54.086696 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:54.086699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:54.086700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:54.086869 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:54.086944 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:54.087339 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 16:34:54.087428 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:34:54.088315 9856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.076685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:33455 (local address 127.23.44.3:39465)
0929 16:34:54.077101 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:54.077139 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:54.078360 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:54.078785 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:54.078790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:54.079092 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:54.079515 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.079527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.080228 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.080230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.081066 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.081075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.081620 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.081629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.081775 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.082462 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:54.082477 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:54.083987 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:54.085887 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:54.085891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:54.085901 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:54.086141 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:54.086461 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:54.086464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:54.086466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:54.086613 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:54.086953 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:54.086957 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:54.087152 (+ 195us) client_negotiation.cc:769] Sending connection context
0929 16:34:54.087294 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 16:34:54.087512 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260929 16:34:54.089370 9854 heartbeater.cc:347] Connected to a master server at 127.23.44.62:33455
I20260929 16:34:54.089607 9854 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:54.090034 9854 heartbeater.cc:511] Master 127.23.44.62:33455 requested a full tablet report, sending...
I20260929 16:34:54.090965 9395 ts_manager.cc:194] Registered new tserver with Master: b903b3379ae6466a9d7c0bf85d000d17 (127.23.44.3:39067)
I20260929 16:34:54.091498 9395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:39465
I20260929 16:34:54.098018 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:34:54.107483 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.099869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.23.44.62:33455)
0929 16:34:54.099997 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:54.100001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:54.100222 (+ 221us) server_negotiation.cc:407] Connection header received
0929 16:34:54.100316 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:54.100318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:54.100372 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:54.100434 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:54.101082 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.101655 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.102473 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.102677 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.102764 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.104340 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:54.104343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:54.104345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:54.104365 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:54.105708 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:54.106262 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:54.106265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:54.106266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:54.106350 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:54.106665 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:54.106668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:54.106669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:54.106869 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:54.106983 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:54.107221 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 16:34:54.107292 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:34:54.110385 9395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:34:54.113250 9395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:34:54.127107 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.121612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:41767 (local address 127.0.0.1:49530)
0929 16:34:54.122345 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:54.122360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:54.122442 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:54.123005 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:54.123008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:54.123038 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:54.123282 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.123289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.124283 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.124289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.125576 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.125587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.126548 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.126552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.126656 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.126691 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:34:54.126828 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 16:34:54.126952 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":72,"spinlock_wait_cycles":2304,"thread_start_us":88,"threads_started":1}
I20260929 16:34:54.127110 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.121987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.23.44.1:41767)
0929 16:34:54.122520 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:54.122539 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:54.122563 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:34:54.122636 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:54.122641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:54.122838 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:54.123031 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:54.123422 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.124152 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.125715 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.126423 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.126550 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.126696 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:54.126865 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 16:34:54.126955 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":114,"threads_started":1}
I20260929 16:34:54.127382 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.121362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:37715 (local address 127.0.0.1:33838)
0929 16:34:54.121975 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:54.121987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:54.122082 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:54.123078 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:54.123081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:54.123102 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:54.123282 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.123289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.124732 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.124735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.125966 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.125976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.126989 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.126992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.127078 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.127099 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:34:54.127151 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 16:34:54.127245 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":80,"thread_start_us":87,"threads_started":1}
I20260929 16:34:54.127785 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.121172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:39067 (local address 127.0.0.1:44752)
0929 16:34:54.121516 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:54.121563 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:54.121677 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:54.122950 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:54.122957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:54.122980 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:54.123186 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.123194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.124767 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.124769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.125920 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.125930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.127157 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.127160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.127471 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.127500 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:34:54.127579 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:34:54.127658 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":308,"spinlock_wait_cycles":896,"thread_start_us":102,"threads_started":1}
I20260929 16:34:54.128098 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.121936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.23.44.2:37715)
0929 16:34:54.122672 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:54.122683 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:54.122701 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:34:54.122766 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:54.122770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:54.122946 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:54.123219 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:54.123380 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.124578 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.126094 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.126845 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.127654 (+ 809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.127791 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:54.127845 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 16:34:54.127942 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":127,"threads_started":1}
I20260929 16:34:54.128593 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.121860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.23.44.3:39067)
0929 16:34:54.122449 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:54.122462 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:54.122490 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:34:54.122570 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:54.122576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:54.122776 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:54.123321 (+ 545us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:54.123482 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.124587 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.126042 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.126845 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.126966 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.127097 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:54.128061 (+ 964us) server_negotiation.cc:299] Negotiation successful
0929 16:34:54.128169 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":131,"threads_started":1}
I20260929 16:34:54.130474 9654 tablet_service.cc:1467] Processing CreateTablet for tablet 508b1d4e5b7a4705b225d1b901247fb9 (DEFAULT_TABLE table=test-table [id=304a9750e4654c57953e1324e0b4301b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:54.130396 9520 tablet_service.cc:1467] Processing CreateTablet for tablet 508b1d4e5b7a4705b225d1b901247fb9 (DEFAULT_TABLE table=test-table [id=304a9750e4654c57953e1324e0b4301b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:54.131383 9654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508b1d4e5b7a4705b225d1b901247fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:54.131426 9520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508b1d4e5b7a4705b225d1b901247fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:54.131167 9788 tablet_service.cc:1467] Processing CreateTablet for tablet 508b1d4e5b7a4705b225d1b901247fb9 (DEFAULT_TABLE table=test-table [id=304a9750e4654c57953e1324e0b4301b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:54.132050 9788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508b1d4e5b7a4705b225d1b901247fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:54.138017 9871 tablet_bootstrap.cc:492] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0: Bootstrap starting.
I20260929 16:34:54.139170 9872 tablet_bootstrap.cc:492] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa: Bootstrap starting.
I20260929 16:34:54.140280 9871 tablet_bootstrap.cc:654] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:54.141196 9872 tablet_bootstrap.cc:654] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:54.141644 9871 log.cc:824] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:54.141762 9873 tablet_bootstrap.cc:492] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17: Bootstrap starting.
I20260929 16:34:54.142398 9872 log.cc:824] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:54.143734 9873 tablet_bootstrap.cc:654] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:54.143775 9871 tablet_bootstrap.cc:492] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0: No bootstrap required, opened a new log
I20260929 16:34:54.143975 9871 ts_tablet_manager.cc:1397] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:34:54.144268 9872 tablet_bootstrap.cc:492] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa: No bootstrap required, opened a new log
I20260929 16:34:54.144446 9872 ts_tablet_manager.cc:1397] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:34:54.144848 9873 log.cc:824] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:54.146342 9873 tablet_bootstrap.cc:492] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17: No bootstrap required, opened a new log
I20260929 16:34:54.146521 9873 ts_tablet_manager.cc:1397] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:34:54.147918 9871 raft_consensus.cc:348] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b903b3379ae6466a9d7c0bf85d000d17" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39067 } } peers { permanent_uuid: "c73b038d6004415cafd93c7471dcbcb0" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 37715 } } peers { permanent_uuid: "4d19460daa874a409e4e98a818371afa" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41767 } }
I20260929 16:34:54.148048 9872 raft_consensus.cc:348] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b903b3379ae6466a9d7c0bf85d000d17" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39067 } } peers { permanent_uuid: "c73b038d6004415cafd93c7471dcbcb0" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 37715 } } peers { permanent_uuid: "4d19460daa874a409e4e98a818371afa" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41767 } }
I20260929 16:34:54.148243 9872 raft_consensus.cc:374] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:54.148231 9871 raft_consensus.cc:374] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:54.148307 9871 raft_consensus.cc:729] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73b038d6004415cafd93c7471dcbcb0, State: Initialized, Role: FOLLOWER
I20260929 16:34:54.148536 9872 raft_consensus.cc:729] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d19460daa874a409e4e98a818371afa, State: Initialized, Role: FOLLOWER
I20260929 16:34:54.148815 9871 consensus_queue.cc:260] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b903b3379ae6466a9d7c0bf85d000d17" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39067 } } peers { permanent_uuid: "c73b038d6004415cafd93c7471dcbcb0" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 37715 } } peers { permanent_uuid: "4d19460daa874a409e4e98a818371afa" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41767 } }
I20260929 16:34:54.148998 9872 consensus_queue.cc:260] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b903b3379ae6466a9d7c0bf85d000d17" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39067 } } peers { permanent_uuid: "c73b038d6004415cafd93c7471dcbcb0" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 37715 } } peers { permanent_uuid: "4d19460daa874a409e4e98a818371afa" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41767 } }
I20260929 16:34:54.149734 9873 raft_consensus.cc:348] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b903b3379ae6466a9d7c0bf85d000d17" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39067 } } peers { permanent_uuid: "c73b038d6004415cafd93c7471dcbcb0" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 37715 } } peers { permanent_uuid: "4d19460daa874a409e4e98a818371afa" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41767 } }
I20260929 16:34:54.149928 9871 ts_tablet_manager.cc:1428] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260929 16:34:54.150068 9872 ts_tablet_manager.cc:1428] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:34:54.150058 9873 raft_consensus.cc:374] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:54.150184 9873 raft_consensus.cc:729] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b903b3379ae6466a9d7c0bf85d000d17, State: Initialized, Role: FOLLOWER
I20260929 16:34:54.150388 9720 heartbeater.cc:503] Master 127.23.44.62:33455 was elected leader, sending a full tablet report...
I20260929 16:34:54.150388 9586 heartbeater.cc:503] Master 127.23.44.62:33455 was elected leader, sending a full tablet report...
I20260929 16:34:54.150668 9873 consensus_queue.cc:260] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b903b3379ae6466a9d7c0bf85d000d17" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39067 } } peers { permanent_uuid: "c73b038d6004415cafd93c7471dcbcb0" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 37715 } } peers { permanent_uuid: "4d19460daa874a409e4e98a818371afa" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41767 } }
I20260929 16:34:54.151352 9854 heartbeater.cc:503] Master 127.23.44.62:33455 was elected leader, sending a full tablet report...
I20260929 16:34:54.151587 9873 ts_tablet_manager.cc:1428] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 16:34:54.159986 9587 tablet.cc:2369] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:34:54.325613 9855 tablet.cc:2369] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:34:54.369979 9721 tablet.cc:2369] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:34:54.406441 9877 raft_consensus.cc:482] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:34:54.406701 9877 raft_consensus.cc:504] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b903b3379ae6466a9d7c0bf85d000d17" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39067 } } peers { permanent_uuid: "c73b038d6004415cafd93c7471dcbcb0" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 37715 } } peers { permanent_uuid: "4d19460daa874a409e4e98a818371afa" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41767 } }
I20260929 16:34:54.407799 9877 leader_election.cc:288] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b903b3379ae6466a9d7c0bf85d000d17 (127.23.44.3:39067), 4d19460daa874a409e4e98a818371afa (127.23.44.1:41767)
I20260929 16:34:54.411994 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.407970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:39067 (local address 127.23.44.2:53589)
0929 16:34:54.408265 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:54.408280 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:54.408434 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:54.408819 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:54.408825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:54.408861 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:54.408990 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.408998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.410022 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.410024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.410848 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.410856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.411563 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.411567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.411669 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.411692 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:34:54.411766 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:34:54.411826 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260929 16:34:54.412015 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.408095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:53589 (local address 127.23.44.3:39067)
0929 16:34:54.408255 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:54.408260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:54.408351 (+ 91us) server_negotiation.cc:407] Connection header received
0929 16:34:54.408561 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:54.408564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:54.408616 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:54.408675 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:54.409154 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.409909 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.410966 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.411439 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.411522 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.411570 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:54.411813 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 16:34:54.411897 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:34:54.412106 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.408409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:58897 (local address 127.23.44.1:41767)
0929 16:34:54.408548 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:54.408555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:54.408570 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:34:54.408629 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:54.408631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:54.408666 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:54.408755 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:54.409070 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.409907 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.411136 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.411553 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.411663 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.411721 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:54.411904 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 16:34:54.411972 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:34:54.412070 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.408068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:41767 (local address 127.23.44.2:58897)
0929 16:34:54.408390 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:54.408403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:54.408504 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:54.408768 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:54.408770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:54.408801 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:54.408946 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.408952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.410059 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.410068 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.411017 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.411027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.411663 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.411665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.411769 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.411794 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:34:54.411870 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:34:54.411930 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":24,"spinlock_wait_cycles":31232,"thread_start_us":58,"threads_started":1}
I20260929 16:34:54.412773 9808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508b1d4e5b7a4705b225d1b901247fb9" candidate_uuid: "c73b038d6004415cafd93c7471dcbcb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b903b3379ae6466a9d7c0bf85d000d17" is_pre_election: true
I20260929 16:34:54.412930 9540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508b1d4e5b7a4705b225d1b901247fb9" candidate_uuid: "c73b038d6004415cafd93c7471dcbcb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d19460daa874a409e4e98a818371afa" is_pre_election: true
I20260929 16:34:54.413062 9808 raft_consensus.cc:2454] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73b038d6004415cafd93c7471dcbcb0 in term 0.
I20260929 16:34:54.413151 9540 raft_consensus.cc:2454] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73b038d6004415cafd93c7471dcbcb0 in term 0.
I20260929 16:34:54.413529 9607 leader_election.cc:302] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b903b3379ae6466a9d7c0bf85d000d17, c73b038d6004415cafd93c7471dcbcb0; no voters:
I20260929 16:34:54.413818 9877 raft_consensus.cc:2781] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:34:54.413887 9877 raft_consensus.cc:482] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:34:54.413918 9877 raft_consensus.cc:3037] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:54.415052 9877 raft_consensus.cc:504] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b903b3379ae6466a9d7c0bf85d000d17" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39067 } } peers { permanent_uuid: "c73b038d6004415cafd93c7471dcbcb0" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 37715 } } peers { permanent_uuid: "4d19460daa874a409e4e98a818371afa" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41767 } }
I20260929 16:34:54.415406 9877 leader_election.cc:288] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [CANDIDATE]: Term 1 election: Requested vote from peers b903b3379ae6466a9d7c0bf85d000d17 (127.23.44.3:39067), 4d19460daa874a409e4e98a818371afa (127.23.44.1:41767)
I20260929 16:34:54.415741 9808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508b1d4e5b7a4705b225d1b901247fb9" candidate_uuid: "c73b038d6004415cafd93c7471dcbcb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b903b3379ae6466a9d7c0bf85d000d17"
I20260929 16:34:54.415853 9808 raft_consensus.cc:3037] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:54.415879 9540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508b1d4e5b7a4705b225d1b901247fb9" candidate_uuid: "c73b038d6004415cafd93c7471dcbcb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d19460daa874a409e4e98a818371afa"
I20260929 16:34:54.416013 9540 raft_consensus.cc:3037] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:54.416848 9808 raft_consensus.cc:2454] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73b038d6004415cafd93c7471dcbcb0 in term 1.
I20260929 16:34:54.417047 9540 raft_consensus.cc:2454] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73b038d6004415cafd93c7471dcbcb0 in term 1.
I20260929 16:34:54.417146 9607 leader_election.cc:302] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b903b3379ae6466a9d7c0bf85d000d17, c73b038d6004415cafd93c7471dcbcb0; no voters:
I20260929 16:34:54.417367 9877 raft_consensus.cc:2781] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:54.417546 9877 raft_consensus.cc:686] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 1 LEADER]: Becoming Leader. State: Replica: c73b038d6004415cafd93c7471dcbcb0, State: Running, Role: LEADER
I20260929 16:34:54.417857 9877 consensus_queue.cc:237] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b903b3379ae6466a9d7c0bf85d000d17" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39067 } } peers { permanent_uuid: "c73b038d6004415cafd93c7471dcbcb0" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 37715 } } peers { permanent_uuid: "4d19460daa874a409e4e98a818371afa" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41767 } }
I20260929 16:34:54.420790 9394 catalog_manager.cc:5495] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c73b038d6004415cafd93c7471dcbcb0 (127.23.44.2). New cstate: current_term: 1 leader_uuid: "c73b038d6004415cafd93c7471dcbcb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b903b3379ae6466a9d7c0bf85d000d17" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c73b038d6004415cafd93c7471dcbcb0" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 37715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d19460daa874a409e4e98a818371afa" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41767 } health_report { overall_health: UNKNOWN } } }
I20260929 16:34:54.487493 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.484180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.23.44.2:37715)
0929 16:34:54.484341 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:54.484345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:54.484361 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:34:54.484426 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:54.484428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:54.484479 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:54.484568 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:54.484806 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.485342 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.486133 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.486340 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.486412 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.486670 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:54.487042 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:54.487154 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:54.487265 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 16:34:54.487329 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:34:54.492969 9540 raft_consensus.cc:1261] T 508b1d4e5b7a4705b225d1b901247fb9 P 4d19460daa874a409e4e98a818371afa [term 1 FOLLOWER]: Refusing update from remote peer c73b038d6004415cafd93c7471dcbcb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:54.492970 9808 raft_consensus.cc:1261] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17 [term 1 FOLLOWER]: Refusing update from remote peer c73b038d6004415cafd93c7471dcbcb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:54.493695 9882 consensus_queue.cc:1035] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b903b3379ae6466a9d7c0bf85d000d17" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:54.493907 9877 consensus_queue.cc:1035] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d19460daa874a409e4e98a818371afa" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:54.508921 9393 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:54.509090 9393 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:54.511474 9393 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42d9ff57002b469cbf0a5f479aa79217: Sending DeleteTablet for 3 replicas of tablet 508b1d4e5b7a4705b225d1b901247fb9
I20260929 16:34:54.512193 9788 tablet_service.cc:1514] Processing DeleteTablet for tablet 508b1d4e5b7a4705b225d1b901247fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:54 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:44752
I20260929 16:34:54.512203 9654 tablet_service.cc:1514] Processing DeleteTablet for tablet 508b1d4e5b7a4705b225d1b901247fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:54 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:33838
I20260929 16:34:54.512436 9520 tablet_service.cc:1514] Processing DeleteTablet for tablet 508b1d4e5b7a4705b225d1b901247fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:54 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:49530
I20260929 16:34:54.512734 9896 tablet_replica.cc:330] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0: stopping tablet replica
I20260929 16:34:54.513031 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 9457
I20260929 16:34:54.513073 9896 raft_consensus.cc:2229] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:34:54.513640 9896 raft_consensus.cc:2258] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:54.514940 9896 ts_tablet_manager.cc:1905] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:54.517436 9896 ts_tablet_manager.cc:1918] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:54.517597 9896 log.cc:1196] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/508b1d4e5b7a4705b225d1b901247fb9
I20260929 16:34:54.517611 9895 tablet_replica.cc:330] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17: stopping tablet replica
I20260929 16:34:54.517871 9895 raft_consensus.cc:2229] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:34:54.518111 9896 ts_tablet_manager.cc:1939] T 508b1d4e5b7a4705b225d1b901247fb9 P c73b038d6004415cafd93c7471dcbcb0: Deleting consensus metadata
I20260929 16:34:54.518196 9895 raft_consensus.cc:2258] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:54.519390 9382 catalog_manager.cc:4841] TS c73b038d6004415cafd93c7471dcbcb0 (127.23.44.2:37715): tablet 508b1d4e5b7a4705b225d1b901247fb9 (table test-table [id=304a9750e4654c57953e1324e0b4301b]) successfully deleted
I20260929 16:34:54.519568 9895 ts_tablet_manager.cc:1905] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:54.522056 9895 ts_tablet_manager.cc:1918] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:54.522167 9895 log.cc:1196] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/508b1d4e5b7a4705b225d1b901247fb9
W20260929 16:34:54.522478 9381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:41767 (error 108)
I20260929 16:34:54.522528 9895 ts_tablet_manager.cc:1939] T 508b1d4e5b7a4705b225d1b901247fb9 P b903b3379ae6466a9d7c0bf85d000d17: Deleting consensus metadata
I20260929 16:34:54.523324 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 9591
I20260929 16:34:54.524219 9379 catalog_manager.cc:4841] TS b903b3379ae6466a9d7c0bf85d000d17 (127.23.44.3:39067): tablet 508b1d4e5b7a4705b225d1b901247fb9 (table test-table [id=304a9750e4654c57953e1324e0b4301b]) successfully deleted
I20260929 16:34:54.524482 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.523192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:41767 (local address 127.0.0.1:49534)
0929 16:34:54.524347 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:54.524411 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1075}
W20260929 16:34:54.524616 9381 catalog_manager.cc:4584] TS 4d19460daa874a409e4e98a818371afa (127.23.44.1:41767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 508b1d4e5b7a4705b225d1b901247fb9: Network error: Client connection negotiation failed: client connection to 127.23.44.1:41767: connect: Connection refused (error 111)
I20260929 16:34:54.530339 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 9725
I20260929 16:34:54.535470 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 9363
2026-09-29T16:34:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2467 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T16:34:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:54Z Disabled control of system clock
I20260929 16:34:54.565064 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:38101
--webserver_interface=127.23.44.62
--webserver_port=0
--builtin_ntp_servers=127.23.44.20:38209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:38101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:34:54.670073 9902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:54.670310 9902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:54.670364 9902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:54.674333 9902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:54.674381 9902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:54.674397 9902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:54.674413 9902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:54.674450 9902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:54.679091 9902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:38101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:38101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:54.680024 9902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:54.681353 9902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:54.689462 9910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:54.689478 9908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:54.689497 9907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:54.689529 9902 server_base.cc:1034] running on GCE node
I20260929 16:34:54.690184 9902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:54.691182 9902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:54.692375 9902 hybrid_clock.cc:648] HybridClock initialized: now 1790699694692359 us; error 26 us; skew 500 ppm
I20260929 16:34:54.694185 9902 webserver.cc:460] Webserver started at http://127.23.44.62:42697/ using document root <none> and password file <none>
I20260929 16:34:54.694725 9902 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:54.694818 9902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:54.695048 9902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:54.696985 9902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "e6929146d2ed4e16aba9d2d7de8788a3"
format_stamp: "Formatted at 2026-09-29 16:34:54 on dist-test-slave-mlmz"
server_key: "bbfc8c94950ae79bd22eff7c5e649f21"
server_key_iv: "ffb050e4c906367bbae1c7983fb0114e"
server_key_version: "encryptionkey@0"
I20260929 16:34:54.697543 9902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "e6929146d2ed4e16aba9d2d7de8788a3"
format_stamp: "Formatted at 2026-09-29 16:34:54 on dist-test-slave-mlmz"
server_key: "bbfc8c94950ae79bd22eff7c5e649f21"
server_key_iv: "ffb050e4c906367bbae1c7983fb0114e"
server_key_version: "encryptionkey@0"
I20260929 16:34:54.701056 9902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 16:34:54.703320 9916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:54.704679 9902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:54.704843 9902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "e6929146d2ed4e16aba9d2d7de8788a3"
format_stamp: "Formatted at 2026-09-29 16:34:54 on dist-test-slave-mlmz"
server_key: "bbfc8c94950ae79bd22eff7c5e649f21"
server_key_iv: "ffb050e4c906367bbae1c7983fb0114e"
server_key_version: "encryptionkey@0"
I20260929 16:34:54.704958 9902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:54.726778 9902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:54.727412 9902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:54.727602 9902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:54.734604 9902 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:38101
I20260929 16:34:54.734627 9968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:38101 every 8 connection(s)
I20260929 16:34:54.735469 9902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:54.738301 9969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:54.740285 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 9902
I20260929 16:34:54.740381 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:34:54.740687 23728 external_mini_cluster.cc:1428] Setting key 91d6a6bebf20cdb1f804d556744eb50b
I20260929 16:34:54.743873 9969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3: Bootstrap starting.
I20260929 16:34:54.745971 9969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:54.746676 9969 log.cc:824] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:54.748672 9969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3: No bootstrap required, opened a new log
I20260929 16:34:54.749465 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.742041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.23.44.62:38101)
0929 16:34:54.742639 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:54.742654 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:54.742685 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:34:54.743305 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:54.743320 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:54.743675 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:54.743998 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:54.744408 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.745395 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.746294 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.746749 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.746924 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.747773 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:54.747778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:54.747789 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:54.748142 (+ 353us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:54.748201 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:54.748221 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:54.748301 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:54.748462 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 16:34:54.748771 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":131,"threads_started":1}
I20260929 16:34:54.752138 9969 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6929146d2ed4e16aba9d2d7de8788a3" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38101 } }
I20260929 16:34:54.752334 9969 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:54.752398 9969 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6929146d2ed4e16aba9d2d7de8788a3, State: Initialized, Role: FOLLOWER
I20260929 16:34:54.752797 9969 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6929146d2ed4e16aba9d2d7de8788a3" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38101 } }
I20260929 16:34:54.752909 9969 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:54.752985 9969 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:54.753099 9969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:54.754136 9969 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6929146d2ed4e16aba9d2d7de8788a3" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38101 } }
I20260929 16:34:54.754456 9969 leader_election.cc:302] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6929146d2ed4e16aba9d2d7de8788a3; no voters:
I20260929 16:34:54.754734 9969 leader_election.cc:288] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:54.754822 9974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:54.755090 9974 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [term 1 LEADER]: Becoming Leader. State: Replica: e6929146d2ed4e16aba9d2d7de8788a3, State: Running, Role: LEADER
I20260929 16:34:54.755414 9974 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6929146d2ed4e16aba9d2d7de8788a3" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38101 } }
I20260929 16:34:54.755861 9969 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:54.756968 9975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6929146d2ed4e16aba9d2d7de8788a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6929146d2ed4e16aba9d2d7de8788a3" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38101 } } }
I20260929 16:34:54.757103 9975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:54.757697 9976 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6929146d2ed4e16aba9d2d7de8788a3. Latest consensus state: current_term: 1 leader_uuid: "e6929146d2ed4e16aba9d2d7de8788a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6929146d2ed4e16aba9d2d7de8788a3" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 38101 } } }
I20260929 16:34:54.757808 9976 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:54.757858 9982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:54.761373 9982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:54.768005 9982 catalog_manager.cc:1347] Generated new cluster ID: cf11e197b3e04e1995219811a1506993
I20260929 16:34:54.768101 9982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:54.778730 9982 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:54.779831 9982 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:54.793479 9982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3: Generated new TSK 0
I20260929 16:34:54.794219 9982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:54.799201 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38101
--builtin_ntp_servers=127.23.44.20:38209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:34:54.922586 9993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:54.922801 9993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:54.922837 9993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:54.926923 9993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:54.926967 9993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:54.927048 9993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:34:54.931583 9993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:54.932474 9993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:54.933732 9993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:54.942479 9998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:54.942565 9993 server_base.cc:1034] running on GCE node
W20260929 16:34:54.942608 9999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:54.942485 10001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:54.943397 9993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:54.944192 9993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:54.945377 9993 hybrid_clock.cc:648] HybridClock initialized: now 1790699694945365 us; error 24 us; skew 500 ppm
I20260929 16:34:54.951709 9993 webserver.cc:460] Webserver started at http://127.23.44.1:38661/ using document root <none> and password file <none>
I20260929 16:34:54.952272 9993 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:54.952363 9993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:54.952585 9993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:54.954564 9993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "d07d2748cd194865b1fab279b78551ea"
format_stamp: "Formatted at 2026-09-29 16:34:54 on dist-test-slave-mlmz"
server_key: "8c9b9a0079fee175c605db926e6a6e11"
server_key_iv: "60727c502f38c1a664162e080fc27a49"
server_key_version: "encryptionkey@0"
I20260929 16:34:54.955124 9993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "d07d2748cd194865b1fab279b78551ea"
format_stamp: "Formatted at 2026-09-29 16:34:54 on dist-test-slave-mlmz"
server_key: "8c9b9a0079fee175c605db926e6a6e11"
server_key_iv: "60727c502f38c1a664162e080fc27a49"
server_key_version: "encryptionkey@0"
I20260929 16:34:54.959092 9993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 16:34:54.961457 10007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:54.962605 9993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:54.962733 9993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "d07d2748cd194865b1fab279b78551ea"
format_stamp: "Formatted at 2026-09-29 16:34:54 on dist-test-slave-mlmz"
server_key: "8c9b9a0079fee175c605db926e6a6e11"
server_key_iv: "60727c502f38c1a664162e080fc27a49"
server_key_version: "encryptionkey@0"
I20260929 16:34:54.962841 9993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:54.979609 9993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:54.980165 9993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:54.980348 9993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:54.980849 9993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:54.981787 9993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:54.981863 9993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:54.981936 9993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:54.981980 9993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:54.990255 9993 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:43957
I20260929 16:34:54.990281 10120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:43957 every 8 connection(s)
I20260929 16:34:54.991030 9993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:34:54.996028 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 9993
I20260929 16:34:54.996155 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:34:54.996443 23728 external_mini_cluster.cc:1428] Setting key a6b1b02a53d4cb5fec2ff1b84440443b
I20260929 16:34:54.998409 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38101
--builtin_ntp_servers=127.23.44.20:38209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:34:55.000437 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.992703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:33401 (local address 127.23.44.62:38101)
0929 16:34:54.992832 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:54.992835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:54.993739 (+ 904us) server_negotiation.cc:407] Connection header received
0929 16:34:54.995085 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:54.995089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:54.995155 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:54.995246 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:54.996194 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.996834 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.998317 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.998624 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.998706 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.999923 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:54.999928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:54.999933 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:54.999947 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:54.999970 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:54.999982 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:55.000062 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:55.000194 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 16:34:55.000257 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:34:55.001111 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:54.992981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38101 (local address 127.23.44.1:33401)
0929 16:34:54.993599 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:54.993635 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:54.994872 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:54.995410 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:54.995417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:54.995492 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:54.996026 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.996047 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.996994 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.996997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.998174 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:54.998186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:54.998778 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:54.998785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:54.999007 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:54.999611 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:34:54.999615 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:54.999670 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:34:54.999755 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:55.000068 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:55.000074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:55.000078 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:34:55.000167 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 16:34:55.000376 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260929 16:34:55.002360 10121 heartbeater.cc:347] Connected to a master server at 127.23.44.62:38101
I20260929 16:34:55.002574 10121 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:55.003022 10121 heartbeater.cc:511] Master 127.23.44.62:38101 requested a full tablet report, sending...
I20260929 16:34:55.004616 9933 ts_manager.cc:194] Registered new tserver with Master: d07d2748cd194865b1fab279b78551ea (127.23.44.1:43957)
I20260929 16:34:55.006307 9933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.44.1:33401
W20260929 16:34:55.104983 10124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:55.105207 10124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:55.105242 10124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:55.109391 10124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:55.109447 10124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:55.109500 10124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:34:55.114173 10124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:55.115018 10124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:55.116214 10124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:55.125674 10129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:55.125869 10124 server_base.cc:1034] running on GCE node
W20260929 16:34:55.125725 10130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:55.125674 10132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:55.126386 10124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:55.126991 10124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:55.128165 10124 hybrid_clock.cc:648] HybridClock initialized: now 1790699695128144 us; error 31 us; skew 500 ppm
I20260929 16:34:55.130112 10124 webserver.cc:460] Webserver started at http://127.23.44.2:45071/ using document root <none> and password file <none>
I20260929 16:34:55.130700 10124 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:55.130767 10124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:55.131011 10124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:55.133039 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "d5fbe7356b2d4eb5b927222a18a1b719"
format_stamp: "Formatted at 2026-09-29 16:34:55 on dist-test-slave-mlmz"
server_key: "c075fe4aeaf33be288c3f47251f8cdad"
server_key_iv: "dd75aabac1c151055fedd9aa38f7f5da"
server_key_version: "encryptionkey@0"
I20260929 16:34:55.133711 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "d5fbe7356b2d4eb5b927222a18a1b719"
format_stamp: "Formatted at 2026-09-29 16:34:55 on dist-test-slave-mlmz"
server_key: "c075fe4aeaf33be288c3f47251f8cdad"
server_key_iv: "dd75aabac1c151055fedd9aa38f7f5da"
server_key_version: "encryptionkey@0"
I20260929 16:34:55.137223 10124 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:55.139678 10138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:55.140889 10124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:55.141028 10124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "d5fbe7356b2d4eb5b927222a18a1b719"
format_stamp: "Formatted at 2026-09-29 16:34:55 on dist-test-slave-mlmz"
server_key: "c075fe4aeaf33be288c3f47251f8cdad"
server_key_iv: "dd75aabac1c151055fedd9aa38f7f5da"
server_key_version: "encryptionkey@0"
I20260929 16:34:55.141144 10124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:55.181067 10124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:55.181910 10124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:55.182125 10124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:55.182803 10124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:55.183940 10124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:55.184021 10124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:55.184092 10124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:55.184154 10124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:55.193360 10124 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:40295
I20260929 16:34:55.193393 10251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:40295 every 8 connection(s)
I20260929 16:34:55.194240 10124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:34:55.195647 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 10124
I20260929 16:34:55.195768 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:34:55.196077 23728 external_mini_cluster.cc:1428] Setting key ea5fd460c0d911c8a2e9de587bd2e787
I20260929 16:34:55.197994 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38101
--builtin_ntp_servers=127.23.44.20:38209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:34:55.204533 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.195938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:38129 (local address 127.23.44.62:38101)
0929 16:34:55.196070 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:55.196076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:55.197315 (+ 1239us) server_negotiation.cc:407] Connection header received
0929 16:34:55.198790 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:55.198796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:55.198856 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:55.198949 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:55.200056 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.200566 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.202433 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.202660 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.202765 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.204047 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:55.204050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:55.204052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:55.204062 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:55.204080 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:55.204088 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:55.204157 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:55.204378 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 16:34:55.204429 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:34:55.205747 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.196370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38101 (local address 127.23.44.2:38129)
0929 16:34:55.197164 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:55.197201 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:55.198596 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:55.199116 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:55.199127 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:55.199243 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:55.199909 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.199929 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.200697 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.200704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.202283 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.202297 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.202836 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.202844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.203076 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.203722 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:34:55.203726 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:55.203784 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:34:55.203882 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:55.204187 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:55.204200 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:55.204210 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 16:34:55.204349 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 16:34:55.204650 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":119,"threads_started":1}
I20260929 16:34:55.206928 10252 heartbeater.cc:347] Connected to a master server at 127.23.44.62:38101
I20260929 16:34:55.207140 10252 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:55.207607 10252 heartbeater.cc:511] Master 127.23.44.62:38101 requested a full tablet report, sending...
I20260929 16:34:55.208587 9933 ts_manager.cc:194] Registered new tserver with Master: d5fbe7356b2d4eb5b927222a18a1b719 (127.23.44.2:40295)
I20260929 16:34:55.209246 9933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.44.2:38129
W20260929 16:34:55.304759 10255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:55.305006 10255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:55.305080 10255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:55.309154 10255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:55.309213 10255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:55.309336 10255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:34:55.313936 10255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:38101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:55.314769 10255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:55.315943 10255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:55.325197 10261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:55.325309 10255 server_base.cc:1034] running on GCE node
W20260929 16:34:55.325199 10260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:55.325212 10263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:55.325815 10255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:55.326364 10255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:55.327572 10255 hybrid_clock.cc:648] HybridClock initialized: now 1790699695327558 us; error 34 us; skew 500 ppm
I20260929 16:34:55.329181 10255 webserver.cc:460] Webserver started at http://127.23.44.3:34845/ using document root <none> and password file <none>
I20260929 16:34:55.329720 10255 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:55.329810 10255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:55.330029 10255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:55.331877 10255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "6c74abee41064df683e983e195138cb8"
format_stamp: "Formatted at 2026-09-29 16:34:55 on dist-test-slave-mlmz"
server_key: "ef9f7afb3ed852daebc20e4717233cc4"
server_key_iv: "7d85077a4fae74336ff25f991d9c54b3"
server_key_version: "encryptionkey@0"
I20260929 16:34:55.332417 10255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "6c74abee41064df683e983e195138cb8"
format_stamp: "Formatted at 2026-09-29 16:34:55 on dist-test-slave-mlmz"
server_key: "ef9f7afb3ed852daebc20e4717233cc4"
server_key_iv: "7d85077a4fae74336ff25f991d9c54b3"
server_key_version: "encryptionkey@0"
I20260929 16:34:55.335763 10255 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 16:34:55.337998 10269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:55.339134 10255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:55.339262 10255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "6c74abee41064df683e983e195138cb8"
format_stamp: "Formatted at 2026-09-29 16:34:55 on dist-test-slave-mlmz"
server_key: "ef9f7afb3ed852daebc20e4717233cc4"
server_key_iv: "7d85077a4fae74336ff25f991d9c54b3"
server_key_version: "encryptionkey@0"
I20260929 16:34:55.339370 10255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:55.359087 10255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:55.359663 10255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:55.359846 10255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:55.360356 10255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:55.361171 10255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:55.361215 10255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:55.361315 10255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:55.361356 10255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:55.369690 10255 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:39785
I20260929 16:34:55.369752 10382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:39785 every 8 connection(s)
I20260929 16:34:55.370523 10255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:34:55.373835 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 10255
I20260929 16:34:55.373965 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:34:55.374217 23728 external_mini_cluster.cc:1428] Setting key c5b550d114f278f0c1e8246d3d0916ee
I20260929 16:34:55.379101 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.372146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:40297 (local address 127.23.44.62:38101)
0929 16:34:55.372274 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:55.372277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:55.373193 (+ 916us) server_negotiation.cc:407] Connection header received
0929 16:34:55.374638 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:55.374641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:55.374702 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:55.374768 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:55.375741 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.376425 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.377550 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.377760 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.377838 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.378737 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:55.378740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:55.378742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:55.378753 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:55.378771 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:55.378778 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:55.378836 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:55.378957 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 16:34:55.379002 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:34:55.379837 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.372488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:38101 (local address 127.23.44.3:40297)
0929 16:34:55.373052 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:55.373087 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:55.374447 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:55.374945 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:55.374951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:55.375067 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:55.375607 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.375622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.376536 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.376538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.377440 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.377449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.377880 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.377882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.378007 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.378515 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:34:55.378516 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:55.378550 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:34:55.378605 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:55.378849 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:55.378854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:55.378859 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:34:55.378942 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:34:55.379123 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260929 16:34:55.380756 10383 heartbeater.cc:347] Connected to a master server at 127.23.44.62:38101
I20260929 16:34:55.380959 10383 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:55.381445 10383 heartbeater.cc:511] Master 127.23.44.62:38101 requested a full tablet report, sending...
I20260929 16:34:55.382354 9933 ts_manager.cc:194] Registered new tserver with Master: 6c74abee41064df683e983e195138cb8 (127.23.44.3:39785)
I20260929 16:34:55.382845 9933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.44.3:40297
I20260929 16:34:55.387601 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:34:55.392663 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.389410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.23.44.62:38101)
0929 16:34:55.389532 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:55.389536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:55.389657 (+ 121us) server_negotiation.cc:407] Connection header received
0929 16:34:55.389742 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:55.389744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:55.389778 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:55.389834 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:55.390036 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.390503 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.391250 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.391449 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.391518 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.392184 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:55.392186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:55.392188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:55.392198 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:55.392214 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:55.392221 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:55.392328 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:55.392448 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 16:34:55.392509 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:34:55.394964 9933 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:34:55.397744 9933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:34:55.411759 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.405707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.23.44.2:40295)
0929 16:34:55.406249 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:55.406263 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:55.406505 (+ 242us) server_negotiation.cc:407] Connection header received
0929 16:34:55.406688 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:55.406694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:55.406911 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:55.407051 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:55.407486 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.408722 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.410405 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.411156 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.411263 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.411437 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:55.411521 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 16:34:55.411608 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":117,"threads_started":1}
I20260929 16:34:55.411752 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.406267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:39785 (local address 127.0.0.1:48434)
0929 16:34:55.406916 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:55.406931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:55.407029 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:55.407386 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:55.407389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:55.407405 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:55.407514 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.407521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.408771 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.408774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.410032 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.410041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.411169 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.411172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.411324 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.411367 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 16:34:55.411478 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:34:55.411609 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":39,"thread_start_us":90,"threads_started":1}
I20260929 16:34:55.411803 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.406541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.23.44.3:39785)
0929 16:34:55.406935 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:55.406949 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:55.406996 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:34:55.407127 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:55.407132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:55.407263 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:55.407406 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:55.407679 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.408633 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.410149 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.410837 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.410938 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.411102 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:55.411533 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 16:34:55.411624 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":138,"threads_started":1}
I20260929 16:34:55.412086 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.405539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.23.44.1:43957)
0929 16:34:55.406122 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:55.406136 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:55.406798 (+ 662us) server_negotiation.cc:407] Connection header received
0929 16:34:55.406868 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:55.406873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:55.407031 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:55.407151 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:55.407486 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.408729 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.410521 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.411015 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.411129 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.411297 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:55.411818 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 16:34:55.411922 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":116,"threads_started":1}
I20260929 16:34:55.412328 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.405437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:43957 (local address 127.0.0.1:60824)
0929 16:34:55.406603 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:55.406619 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:55.406689 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:55.407291 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:55.407293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:55.407305 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:55.407365 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.407370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.409196 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.409199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.410399 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.410407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.411140 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.411144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.411581 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.411600 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 16:34:55.411664 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:34:55.411731 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":344,"thread_start_us":64,"threads_started":1}
I20260929 16:34:55.412662 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.405563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:40295 (local address 127.0.0.1:33566)
0929 16:34:55.406332 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:55.406378 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:55.406505 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:55.407217 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:55.407222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:55.407244 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:55.407360 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.407366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.408925 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.408927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.410279 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.410289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.411292 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.411295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.411378 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.411405 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:34:55.411490 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:34:55.412533 (+ 1043us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":983,"thread_start_us":87,"threads_started":1}
I20260929 16:34:55.414803 10055 tablet_service.cc:1467] Processing CreateTablet for tablet d24c6b4f5820468c9951ef2ae130848e (DEFAULT_TABLE table=test-table [id=5c76c43476454f4b84deda4cc2d9666a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:55.414896 10317 tablet_service.cc:1467] Processing CreateTablet for tablet d24c6b4f5820468c9951ef2ae130848e (DEFAULT_TABLE table=test-table [id=5c76c43476454f4b84deda4cc2d9666a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:55.414935 10186 tablet_service.cc:1467] Processing CreateTablet for tablet d24c6b4f5820468c9951ef2ae130848e (DEFAULT_TABLE table=test-table [id=5c76c43476454f4b84deda4cc2d9666a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:55.415712 10055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24c6b4f5820468c9951ef2ae130848e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:55.415841 10317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24c6b4f5820468c9951ef2ae130848e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:55.415841 10186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24c6b4f5820468c9951ef2ae130848e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:55.421244 10400 tablet_bootstrap.cc:492] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea: Bootstrap starting.
I20260929 16:34:55.422933 10402 tablet_bootstrap.cc:492] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719: Bootstrap starting.
I20260929 16:34:55.423904 10400 tablet_bootstrap.cc:654] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:55.424583 10402 tablet_bootstrap.cc:654] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:55.424587 10400 log.cc:824] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:55.425297 10402 log.cc:824] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:55.425572 10401 tablet_bootstrap.cc:492] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8: Bootstrap starting.
I20260929 16:34:55.426409 10400 tablet_bootstrap.cc:492] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea: No bootstrap required, opened a new log
I20260929 16:34:55.426592 10400 ts_tablet_manager.cc:1397] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 16:34:55.427210 10402 tablet_bootstrap.cc:492] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719: No bootstrap required, opened a new log
I20260929 16:34:55.427383 10402 ts_tablet_manager.cc:1397] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:34:55.427464 10401 tablet_bootstrap.cc:654] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:55.428090 10401 log.cc:824] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:55.429606 10401 tablet_bootstrap.cc:492] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8: No bootstrap required, opened a new log
I20260929 16:34:55.429801 10401 ts_tablet_manager.cc:1397] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 16:34:55.431550 10400 raft_consensus.cc:348] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d07d2748cd194865b1fab279b78551ea" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43957 } } peers { permanent_uuid: "d5fbe7356b2d4eb5b927222a18a1b719" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 40295 } } peers { permanent_uuid: "6c74abee41064df683e983e195138cb8" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39785 } }
I20260929 16:34:55.431550 10402 raft_consensus.cc:348] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d07d2748cd194865b1fab279b78551ea" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43957 } } peers { permanent_uuid: "d5fbe7356b2d4eb5b927222a18a1b719" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 40295 } } peers { permanent_uuid: "6c74abee41064df683e983e195138cb8" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39785 } }
I20260929 16:34:55.431806 10402 raft_consensus.cc:374] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:55.431806 10400 raft_consensus.cc:374] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:55.431864 10402 raft_consensus.cc:729] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5fbe7356b2d4eb5b927222a18a1b719, State: Initialized, Role: FOLLOWER
I20260929 16:34:55.431871 10400 raft_consensus.cc:729] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d07d2748cd194865b1fab279b78551ea, State: Initialized, Role: FOLLOWER
I20260929 16:34:55.432301 10400 consensus_queue.cc:260] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d07d2748cd194865b1fab279b78551ea" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43957 } } peers { permanent_uuid: "d5fbe7356b2d4eb5b927222a18a1b719" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 40295 } } peers { permanent_uuid: "6c74abee41064df683e983e195138cb8" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39785 } }
I20260929 16:34:55.432301 10402 consensus_queue.cc:260] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d07d2748cd194865b1fab279b78551ea" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43957 } } peers { permanent_uuid: "d5fbe7356b2d4eb5b927222a18a1b719" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 40295 } } peers { permanent_uuid: "6c74abee41064df683e983e195138cb8" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39785 } }
I20260929 16:34:55.432928 10252 heartbeater.cc:503] Master 127.23.44.62:38101 was elected leader, sending a full tablet report...
I20260929 16:34:55.433208 10121 heartbeater.cc:503] Master 127.23.44.62:38101 was elected leader, sending a full tablet report...
I20260929 16:34:55.433075 10401 raft_consensus.cc:348] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d07d2748cd194865b1fab279b78551ea" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43957 } } peers { permanent_uuid: "d5fbe7356b2d4eb5b927222a18a1b719" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 40295 } } peers { permanent_uuid: "6c74abee41064df683e983e195138cb8" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39785 } }
I20260929 16:34:55.433336 10401 raft_consensus.cc:374] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:55.433398 10401 raft_consensus.cc:729] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c74abee41064df683e983e195138cb8, State: Initialized, Role: FOLLOWER
I20260929 16:34:55.433831 10401 consensus_queue.cc:260] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d07d2748cd194865b1fab279b78551ea" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43957 } } peers { permanent_uuid: "d5fbe7356b2d4eb5b927222a18a1b719" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 40295 } } peers { permanent_uuid: "6c74abee41064df683e983e195138cb8" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39785 } }
I20260929 16:34:55.434250 10402 ts_tablet_manager.cc:1428] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 16:34:55.434746 10401 ts_tablet_manager.cc:1428] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:34:55.435079 10383 heartbeater.cc:503] Master 127.23.44.62:38101 was elected leader, sending a full tablet report...
I20260929 16:34:55.436614 10400 ts_tablet_manager.cc:1428] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
W20260929 16:34:55.445142 10253 tablet.cc:2369] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:34:55.492121 10122 tablet.cc:2369] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:34:55.510737 10407 raft_consensus.cc:482] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:34:55.510937 10407 raft_consensus.cc:504] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d07d2748cd194865b1fab279b78551ea" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43957 } } peers { permanent_uuid: "d5fbe7356b2d4eb5b927222a18a1b719" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 40295 } } peers { permanent_uuid: "6c74abee41064df683e983e195138cb8" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39785 } }
I20260929 16:34:55.511996 10407 leader_election.cc:288] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5fbe7356b2d4eb5b927222a18a1b719 (127.23.44.2:40295), 6c74abee41064df683e983e195138cb8 (127.23.44.3:39785)
I20260929 16:34:55.515812 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.512345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:50897 (local address 127.23.44.2:40295)
0929 16:34:55.512472 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:55.512477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:55.512628 (+ 151us) server_negotiation.cc:407] Connection header received
0929 16:34:55.512786 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:55.512788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:55.512840 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:55.512918 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:55.513234 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.513773 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.514858 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.515246 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.515370 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.515426 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:55.515625 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 16:34:55.515688 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:34:55.515841 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.512345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:33109 (local address 127.23.44.3:39785)
0929 16:34:55.512533 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:55.512539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:55.512678 (+ 139us) server_negotiation.cc:407] Connection header received
0929 16:34:55.512837 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:55.512842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:55.512888 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:55.512972 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:55.513250 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.513775 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.514878 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.515271 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.515386 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.515441 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:55.515651 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 16:34:55.515715 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 16:34:55.515864 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.512246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:39785 (local address 127.23.44.1:33109)
0929 16:34:55.512592 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:55.512603 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:55.512723 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:55.512980 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:55.512983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:55.512999 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:55.513136 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.513141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.513884 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.513889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.514758 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.514766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.515392 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.515395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.515530 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.515547 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 16:34:55.515634 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:34:55.515711 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":50,"thread_start_us":85,"threads_started":1}
I20260929 16:34:55.516036 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.512227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:40295 (local address 127.23.44.1:50897)
0929 16:34:55.512538 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:55.512550 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:55.512671 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:55.512938 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:55.512941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:55.512956 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:55.513119 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.513125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.513881 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.513886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.514728 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:55.514736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.515382 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.515385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:55.515496 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.515526 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:34:55.515609 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:34:55.515678 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":74,"thread_start_us":95,"threads_started":1}
I20260929 16:34:55.516516 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24c6b4f5820468c9951ef2ae130848e" candidate_uuid: "d07d2748cd194865b1fab279b78551ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c74abee41064df683e983e195138cb8" is_pre_election: true
I20260929 16:34:55.516633 10206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24c6b4f5820468c9951ef2ae130848e" candidate_uuid: "d07d2748cd194865b1fab279b78551ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5fbe7356b2d4eb5b927222a18a1b719" is_pre_election: true
I20260929 16:34:55.516811 10337 raft_consensus.cc:2454] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d07d2748cd194865b1fab279b78551ea in term 0.
I20260929 16:34:55.516959 10206 raft_consensus.cc:2454] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d07d2748cd194865b1fab279b78551ea in term 0.
I20260929 16:34:55.517370 10009 leader_election.cc:302] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6c74abee41064df683e983e195138cb8, d07d2748cd194865b1fab279b78551ea, d5fbe7356b2d4eb5b927222a18a1b719; no voters:
I20260929 16:34:55.517669 10407 raft_consensus.cc:2781] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:34:55.517742 10407 raft_consensus.cc:482] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:34:55.517772 10407 raft_consensus.cc:3037] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:55.518733 10407 raft_consensus.cc:504] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d07d2748cd194865b1fab279b78551ea" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43957 } } peers { permanent_uuid: "d5fbe7356b2d4eb5b927222a18a1b719" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 40295 } } peers { permanent_uuid: "6c74abee41064df683e983e195138cb8" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39785 } }
I20260929 16:34:55.519093 10407 leader_election.cc:288] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [CANDIDATE]: Term 1 election: Requested vote from peers d5fbe7356b2d4eb5b927222a18a1b719 (127.23.44.2:40295), 6c74abee41064df683e983e195138cb8 (127.23.44.3:39785)
I20260929 16:34:55.519416 10206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24c6b4f5820468c9951ef2ae130848e" candidate_uuid: "d07d2748cd194865b1fab279b78551ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5fbe7356b2d4eb5b927222a18a1b719"
I20260929 16:34:55.519482 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24c6b4f5820468c9951ef2ae130848e" candidate_uuid: "d07d2748cd194865b1fab279b78551ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c74abee41064df683e983e195138cb8"
I20260929 16:34:55.519523 10206 raft_consensus.cc:3037] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:55.519619 10337 raft_consensus.cc:3037] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:55.520496 10206 raft_consensus.cc:2454] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d07d2748cd194865b1fab279b78551ea in term 1.
I20260929 16:34:55.520650 10337 raft_consensus.cc:2454] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d07d2748cd194865b1fab279b78551ea in term 1.
I20260929 16:34:55.520797 10011 leader_election.cc:302] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d07d2748cd194865b1fab279b78551ea, d5fbe7356b2d4eb5b927222a18a1b719; no voters:
I20260929 16:34:55.520967 10407 raft_consensus.cc:2781] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:55.521173 10407 raft_consensus.cc:686] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [term 1 LEADER]: Becoming Leader. State: Replica: d07d2748cd194865b1fab279b78551ea, State: Running, Role: LEADER
I20260929 16:34:55.521503 10407 consensus_queue.cc:237] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d07d2748cd194865b1fab279b78551ea" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43957 } } peers { permanent_uuid: "d5fbe7356b2d4eb5b927222a18a1b719" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 40295 } } peers { permanent_uuid: "6c74abee41064df683e983e195138cb8" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39785 } }
I20260929 16:34:55.524408 9932 catalog_manager.cc:5495] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea reported cstate change: term changed from 0 to 1, leader changed from <none> to d07d2748cd194865b1fab279b78551ea (127.23.44.1). New cstate: current_term: 1 leader_uuid: "d07d2748cd194865b1fab279b78551ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d07d2748cd194865b1fab279b78551ea" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 43957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5fbe7356b2d4eb5b927222a18a1b719" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c74abee41064df683e983e195138cb8" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39785 } health_report { overall_health: UNKNOWN } } }
I20260929 16:34:55.563413 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:55.559934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.23.44.1:43957)
0929 16:34:55.560060 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:55.560064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:55.560077 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:34:55.560163 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:55.560165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:55.560215 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:55.560299 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:55.560549 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.561050 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.561997 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:55.562197 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:55.562271 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:55.562519 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:55.562881 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:55.562983 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:55.563269 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:34:55.563303 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:34:55.569248 10206 raft_consensus.cc:1261] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719 [term 1 FOLLOWER]: Refusing update from remote peer d07d2748cd194865b1fab279b78551ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:55.569248 10337 raft_consensus.cc:1261] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8 [term 1 FOLLOWER]: Refusing update from remote peer d07d2748cd194865b1fab279b78551ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:55.569983 10411 consensus_queue.cc:1035] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c74abee41064df683e983e195138cb8" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:55.570153 10407 consensus_queue.cc:1035] T d24c6b4f5820468c9951ef2ae130848e P d07d2748cd194865b1fab279b78551ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5fbe7356b2d4eb5b927222a18a1b719" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:55.576166 10414 mvcc.cc:205] Tried to move back new op lower bound from 7334705953044221952 to 7334705952858759168. Current Snapshot: MvccSnapshot[applied={T|T < 7334705953044221952}]
I20260929 16:34:55.576479 10416 mvcc.cc:205] Tried to move back new op lower bound from 7334705953044221952 to 7334705952858759168. Current Snapshot: MvccSnapshot[applied={T|T < 7334705953044221952}]
I20260929 16:34:55.586537 9932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:55.586716 9932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:55.589304 9932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6929146d2ed4e16aba9d2d7de8788a3: Sending DeleteTablet for 3 replicas of tablet d24c6b4f5820468c9951ef2ae130848e
I20260929 16:34:55.590025 10186 tablet_service.cc:1514] Processing DeleteTablet for tablet d24c6b4f5820468c9951ef2ae130848e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:55 UTC) from {username='slave'} at 127.0.0.1:33566
I20260929 16:34:55.590082 10055 tablet_service.cc:1514] Processing DeleteTablet for tablet d24c6b4f5820468c9951ef2ae130848e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:55 UTC) from {username='slave'} at 127.0.0.1:60824
I20260929 16:34:55.590216 10317 tablet_service.cc:1514] Processing DeleteTablet for tablet d24c6b4f5820468c9951ef2ae130848e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:55 UTC) from {username='slave'} at 127.0.0.1:48434
I20260929 16:34:55.590600 10424 tablet_replica.cc:330] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719: stopping tablet replica
I20260929 16:34:55.590926 10424 raft_consensus.cc:2229] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:34:55.591009 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 9993
I20260929 16:34:55.591204 10424 raft_consensus.cc:2258] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:55.593179 10424 ts_tablet_manager.cc:1905] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:55.593180 10426 tablet_replica.cc:330] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8: stopping tablet replica
I20260929 16:34:55.593526 10426 raft_consensus.cc:2229] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:34:55.593822 10426 raft_consensus.cc:2258] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:55.595054 10426 ts_tablet_manager.cc:1905] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:55.595873 10424 ts_tablet_manager.cc:1918] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:55.595965 10424 log.cc:1196] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/d24c6b4f5820468c9951ef2ae130848e
I20260929 16:34:55.596268 10424 ts_tablet_manager.cc:1939] T d24c6b4f5820468c9951ef2ae130848e P d5fbe7356b2d4eb5b927222a18a1b719: Deleting consensus metadata
I20260929 16:34:55.597436 9920 catalog_manager.cc:4841] TS d5fbe7356b2d4eb5b927222a18a1b719 (127.23.44.2:40295): tablet d24c6b4f5820468c9951ef2ae130848e (table test-table [id=5c76c43476454f4b84deda4cc2d9666a]) successfully deleted
I20260929 16:34:55.597764 10426 ts_tablet_manager.cc:1918] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:55.597832 10426 log.cc:1196] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/d24c6b4f5820468c9951ef2ae130848e
I20260929 16:34:55.598110 10426 ts_tablet_manager.cc:1939] T d24c6b4f5820468c9951ef2ae130848e P 6c74abee41064df683e983e195138cb8: Deleting consensus metadata
W20260929 16:34:55.598929 9918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:43957 (error 108)
I20260929 16:34:55.599344 9918 catalog_manager.cc:4841] TS 6c74abee41064df683e983e195138cb8 (127.23.44.3:39785): tablet d24c6b4f5820468c9951ef2ae130848e (table test-table [id=5c76c43476454f4b84deda4cc2d9666a]) successfully deleted
W20260929 16:34:55.599994 10395 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:55.599633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:43957 (local address 127.0.0.1:60846)
0929 16:34:55.599767 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:55.599786 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:55.599865 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:43957: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:34:55.600159 9918 catalog_manager.cc:4584] TS d07d2748cd194865b1fab279b78551ea (127.23.44.1:43957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d24c6b4f5820468c9951ef2ae130848e: Network error: Client connection negotiation failed: client connection to 127.23.44.1:43957: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260929 16:34:55.600191 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 10124
I20260929 16:34:55.606287 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 10255
I20260929 16:34:55.612392 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 9902
2026-09-29T16:34:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1075 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:55 dist-test-slave-mlmz krb5kdc[10430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:55 dist-test-slave-mlmz krb5kdc[10430](info): set up 2 sockets
Sep 29 16:34:55 dist-test-slave-mlmz krb5kdc[10430](info): commencing operation
krb5kdc: starting...
W20260929 16:34:56.653599 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:34:56 dist-test-slave-mlmz krb5kdc[10430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:34:56.768309 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:33025
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:33025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:56.872323 10443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:56.872594 10443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:56.872670 10443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:56.876578 10443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:56.876659 10443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:56.876703 10443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:56.876760 10443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:56.876796 10443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:56.881765 10443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:33025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:56.882683 10443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:56.883608 10443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:56.891498 10451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:56.891515 10448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:56.891515 10449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:56.891657 10443 server_base.cc:1034] running on GCE node
I20260929 16:34:56.892382 10443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:56.893481 10443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:56.894665 10443 hybrid_clock.cc:648] HybridClock initialized: now 1790699696894652 us; error 27 us; skew 500 ppm
Sep 29 16:34:56 dist-test-slave-mlmz krb5kdc[10430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699696, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:56.897306 10443 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:34:56.898115 10443 webserver.cc:460] Webserver started at http://127.23.44.62:37607/ using document root <none> and password file <none>
I20260929 16:34:56.898617 10443 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:56.898675 10443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:56.898826 10443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:56.900674 10443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "c1d85b61a7e34c3a8a7c955337afe16f"
format_stamp: "Formatted at 2026-09-29 16:34:56 on dist-test-slave-mlmz"
server_key: "2e7453e321c5179f75f7b0aa6929a656"
server_key_iv: "3a37d1e6507992aab19139c359e6ab1c"
server_key_version: "encryptionkey@0"
I20260929 16:34:56.901216 10443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "c1d85b61a7e34c3a8a7c955337afe16f"
format_stamp: "Formatted at 2026-09-29 16:34:56 on dist-test-slave-mlmz"
server_key: "2e7453e321c5179f75f7b0aa6929a656"
server_key_iv: "3a37d1e6507992aab19139c359e6ab1c"
server_key_version: "encryptionkey@0"
I20260929 16:34:56.904623 10443 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 16:34:56.906961 10458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:56.908310 10443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:34:56.908416 10443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "c1d85b61a7e34c3a8a7c955337afe16f"
format_stamp: "Formatted at 2026-09-29 16:34:56 on dist-test-slave-mlmz"
server_key: "2e7453e321c5179f75f7b0aa6929a656"
server_key_iv: "3a37d1e6507992aab19139c359e6ab1c"
server_key_version: "encryptionkey@0"
I20260929 16:34:56.908499 10443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:56.922276 10443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:56.925033 10443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:56.925169 10443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:56.931768 10443 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:33025
I20260929 16:34:56.931792 10510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:33025 every 8 connection(s)
I20260929 16:34:56.932616 10443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:56.933588 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 10443
I20260929 16:34:56.933738 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:34:56.934013 23728 external_mini_cluster.cc:1428] Setting key 045e79c90bef3db55fdd9a8043038c7c
I20260929 16:34:56.935418 10511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:56.941795 10511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f: Bootstrap starting.
Sep 29 16:34:56 dist-test-slave-mlmz krb5kdc[10430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:56.944229 10511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:56.945124 10511 log.cc:824] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:56.947867 10511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f: No bootstrap required, opened a new log
I20260929 16:34:56.949074 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:56.935344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.23.44.62:33025)
0929 16:34:56.935759 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:56.935777 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:56.935817 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:34:56.936593 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:56.936613 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:56.936995 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:56.937365 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:56.938003 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:56.939456 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:56.940290 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:56.940979 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:56.941136 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:56.943764 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:56.943770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:56.943784 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:56.943813 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:56.945747 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:56.946413 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:56.946422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:56.946432 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:56.946579 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:56.946866 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:56.946869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:56.946871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:56.947424 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:56.947631 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:56.947824 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 16:34:56.948123 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":152,"threads_started":1}
I20260929 16:34:56.952291 10511 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d85b61a7e34c3a8a7c955337afe16f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 33025 } }
I20260929 16:34:56.952512 10511 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:56.952548 10511 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d85b61a7e34c3a8a7c955337afe16f, State: Initialized, Role: FOLLOWER
I20260929 16:34:56.953001 10511 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d85b61a7e34c3a8a7c955337afe16f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 33025 } }
I20260929 16:34:56.953125 10511 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:56.953171 10511 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:56.953243 10511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:56.954304 10511 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d85b61a7e34c3a8a7c955337afe16f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 33025 } }
I20260929 16:34:56.954672 10511 leader_election.cc:302] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1d85b61a7e34c3a8a7c955337afe16f; no voters:
I20260929 16:34:56.954979 10511 leader_election.cc:288] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:56.955114 10516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:56.955376 10516 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [term 1 LEADER]: Becoming Leader. State: Replica: c1d85b61a7e34c3a8a7c955337afe16f, State: Running, Role: LEADER
I20260929 16:34:56.955672 10516 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d85b61a7e34c3a8a7c955337afe16f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 33025 } }
I20260929 16:34:56.956192 10511 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:56.957297 10518 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1d85b61a7e34c3a8a7c955337afe16f. Latest consensus state: current_term: 1 leader_uuid: "c1d85b61a7e34c3a8a7c955337afe16f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d85b61a7e34c3a8a7c955337afe16f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 33025 } } }
I20260929 16:34:56.957440 10518 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:56.957793 10517 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1d85b61a7e34c3a8a7c955337afe16f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d85b61a7e34c3a8a7c955337afe16f" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 33025 } } }
I20260929 16:34:56.957898 10517 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:56.957847 10525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:56.961155 10525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:56.966156 10525 catalog_manager.cc:1347] Generated new cluster ID: b0e181ad10cc47d9b25df7563cff34f3
I20260929 16:34:56.966219 10525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:56.980394 10525 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:56.981208 10525 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:56.986289 10525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f: Generated new TSK 0
I20260929 16:34:56.986861 10525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.44.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
Entry for principal kudu/127.23.44.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab.
I20260929 16:34:57.024335 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:33025
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:57.129076 10537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:57.129323 10537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:57.129400 10537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:57.133323 10537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:57.133370 10537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:57.133472 10537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:34:57.138453 10537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.1.keytab
--principal=kudu/127.23.44.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:57.139354 10537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:57.140277 10537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:57.149286 10542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:57.149312 10543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:57.149310 10545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:57.149359 10537 server_base.cc:1034] running on GCE node
I20260929 16:34:57.149852 10537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:57.150424 10537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:57.151590 10537 hybrid_clock.cc:648] HybridClock initialized: now 1790699697151574 us; error 29 us; skew 500 ppm
Sep 29 16:34:57 dist-test-slave-mlmz krb5kdc[10430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699697, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:57.155026 10537 init.cc:377] Logged in from keytab as kudu/127.23.44.1@KRBTEST.COM (short username kudu)
I20260929 16:34:57.156128 10537 webserver.cc:460] Webserver started at http://127.23.44.1:37199/ using document root <none> and password file <none>
I20260929 16:34:57.156862 10537 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:57.156951 10537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:57.157178 10537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:57.159912 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "519322c5a3634ab091de767b07dda313"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-mlmz"
server_key: "9b3b8911e8230fb237693a01323999ea"
server_key_iv: "06fc14bd84921d568b96e74016c0baae"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.160552 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "519322c5a3634ab091de767b07dda313"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-mlmz"
server_key: "9b3b8911e8230fb237693a01323999ea"
server_key_iv: "06fc14bd84921d568b96e74016c0baae"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.164062 10537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:34:57.166347 10552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:57.167541 10537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:34:57.167699 10537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "519322c5a3634ab091de767b07dda313"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-mlmz"
server_key: "9b3b8911e8230fb237693a01323999ea"
server_key_iv: "06fc14bd84921d568b96e74016c0baae"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.167807 10537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:57.186888 10537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:57.189520 10537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:57.189706 10537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:57.190254 10537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:57.191265 10537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:57.191339 10537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:57.191402 10537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:57.191434 10537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:57.200362 10537 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:41271
I20260929 16:34:57.200402 10665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:41271 every 8 connection(s)
I20260929 16:34:57.201145 10537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:34:57.210251 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 10537
I20260929 16:34:57.210408 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
Sep 29 16:34:57 dist-test-slave-mlmz krb5kdc[10430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699697, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.1@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:57.210788 23728 external_mini_cluster.cc:1428] Setting key b111a33bc20925981d43102b1813b3c0
I20260929 16:34:57.215775 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.202859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:41179 (local address 127.23.44.62:33025)
0929 16:34:57.202987 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:57.202991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:57.203765 (+ 774us) server_negotiation.cc:407] Connection header received
0929 16:34:57.205175 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:57.205179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:57.205249 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:57.205391 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:57.206524 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.207284 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.208384 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.208646 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.208739 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.211437 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:57.211446 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:57.211448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:57.211471 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:57.213517 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:57.213980 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:57.213987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:57.213991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:57.214101 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:57.214499 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:57.214502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:57.214503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:57.214672 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:57.214758 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:57.215368 (+ 610us) server_negotiation.cc:299] Negotiation successful
0929 16:34:57.215513 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:34:57.216308 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.203176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:33025 (local address 127.23.44.1:41179)
0929 16:34:57.203611 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:57.203653 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:57.204958 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:57.205559 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:57.205566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:57.205914 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:57.206369 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.206383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.207421 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.207425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.208264 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.208273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.208785 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.208788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.208917 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.209512 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:57.209527 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:57.211264 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:57.213629 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:57.213633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:57.213642 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:57.213878 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:57.214216 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:57.214222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:57.214225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:57.214401 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:57.214779 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:57.214788 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:57.215084 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 16:34:57.215296 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 16:34:57.215554 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
I20260929 16:34:57.217563 10666 heartbeater.cc:347] Connected to a master server at 127.23.44.62:33025
I20260929 16:34:57.217826 10666 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:57.218443 10666 heartbeater.cc:511] Master 127.23.44.62:33025 requested a full tablet report, sending...
I20260929 16:34:57.220163 10475 ts_manager.cc:194] Registered new tserver with Master: 519322c5a3634ab091de767b07dda313 (127.23.44.1:41271)
WARNING: no policy specified for kudu/127.23.44.2@KRBTEST.COM; defaulting to no policy
I20260929 16:34:57.221948 10475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.1@KRBTEST.COM'} at 127.23.44.1:41179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
Entry for principal kudu/127.23.44.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab.
I20260929 16:34:57.240293 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:33025
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:57.343626 10671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:57.343886 10671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:57.343955 10671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:57.348129 10671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:57.348208 10671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:57.348296 10671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:34:57.353247 10671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.2.keytab
--principal=kudu/127.23.44.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:57.354189 10671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:57.355082 10671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:57.363999 10676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:57.364009 10677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:57.364070 10679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:57.364388 10671 server_base.cc:1034] running on GCE node
I20260929 16:34:57.364771 10671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:57.365450 10671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:57.366627 10671 hybrid_clock.cc:648] HybridClock initialized: now 1790699697366617 us; error 26 us; skew 500 ppm
Sep 29 16:34:57 dist-test-slave-mlmz krb5kdc[10430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699697, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:57.369215 10671 init.cc:377] Logged in from keytab as kudu/127.23.44.2@KRBTEST.COM (short username kudu)
I20260929 16:34:57.370033 10671 webserver.cc:460] Webserver started at http://127.23.44.2:36385/ using document root <none> and password file <none>
I20260929 16:34:57.370548 10671 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:57.370635 10671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:57.370857 10671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:57.372864 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "fd9318bbde8a45b79b76decd273c2a40"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-mlmz"
server_key: "bbcaccc3f9f9982291ec0d59548f59ab"
server_key_iv: "44a52a8d658fcdf3ab7820e0875fbff8"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.373442 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "fd9318bbde8a45b79b76decd273c2a40"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-mlmz"
server_key: "bbcaccc3f9f9982291ec0d59548f59ab"
server_key_iv: "44a52a8d658fcdf3ab7820e0875fbff8"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.376770 10671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:57.379014 10686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:57.380075 10671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:34:57.380208 10671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "fd9318bbde8a45b79b76decd273c2a40"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-mlmz"
server_key: "bbcaccc3f9f9982291ec0d59548f59ab"
server_key_iv: "44a52a8d658fcdf3ab7820e0875fbff8"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.380309 10671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:57.395499 10671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:57.398440 10671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:57.398633 10671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:57.399139 10671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:57.399855 10671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:57.399923 10671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:57.399987 10671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:57.400033 10671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:57.408425 10671 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:36177
I20260929 16:34:57.408447 10799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:36177 every 8 connection(s)
I20260929 16:34:57.409184 10671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:34:57.415632 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 10671
I20260929 16:34:57.415741 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:34:57.416057 23728 external_mini_cluster.cc:1428] Setting key 91e0e6e9d3d3b208bbc627737ea57381
Sep 29 16:34:57 dist-test-slave-mlmz krb5kdc[10430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699697, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.2@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:57.423339 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.410867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:49953 (local address 127.23.44.62:33025)
0929 16:34:57.410989 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:57.410994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:57.411660 (+ 666us) server_negotiation.cc:407] Connection header received
0929 16:34:57.413143 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:57.413146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:57.413236 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:57.413324 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:57.414672 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.415216 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.416446 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.416681 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.416803 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.419393 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:57.419400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:57.419403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:57.419434 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:57.421105 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:57.421706 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:57.421713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:57.421717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:57.421820 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:57.422145 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:57.422151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:57.422156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:57.422399 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:57.422513 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:57.422968 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 16:34:57.423109 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:34:57.423870 10802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.411103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:33025 (local address 127.23.44.2:49953)
0929 16:34:57.411527 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:57.411565 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:57.412836 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:57.413570 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:57.413576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:57.414100 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:57.414522 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.414535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.415333 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.415336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.416324 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.416332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.416833 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.416836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.416996 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.417591 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:57.417606 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:57.419231 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:57.421222 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:57.421226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:57.421236 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:57.421594 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:57.421918 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:57.421919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:57.421921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:57.422040 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:57.422514 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:57.422519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:57.422725 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 16:34:57.422863 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 16:34:57.423088 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":152,"threads_started":1}
I20260929 16:34:57.424959 10800 heartbeater.cc:347] Connected to a master server at 127.23.44.62:33025
I20260929 16:34:57.425194 10800 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.44.3@KRBTEST.COM; defaulting to no policy
I20260929 16:34:57.425709 10800 heartbeater.cc:511] Master 127.23.44.62:33025 requested a full tablet report, sending...
I20260929 16:34:57.426644 10475 ts_manager.cc:194] Registered new tserver with Master: fd9318bbde8a45b79b76decd273c2a40 (127.23.44.2:36177)
I20260929 16:34:57.427297 10475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.2@KRBTEST.COM'} at 127.23.44.2:49953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
Entry for principal kudu/127.23.44.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab.
I20260929 16:34:57.442333 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:33025
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:57.547731 10805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:57.547983 10805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:57.548058 10805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:57.551998 10805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:57.552083 10805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:57.552174 10805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:34:57.557443 10805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.3.keytab
--principal=kudu/127.23.44.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:57.558467 10805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:57.559408 10805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:57.568509 10811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:57.568578 10813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:57.568509 10810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:57.568748 10805 server_base.cc:1034] running on GCE node
I20260929 16:34:57.569229 10805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:57.569923 10805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:57.571100 10805 hybrid_clock.cc:648] HybridClock initialized: now 1790699697571081 us; error 31 us; skew 500 ppm
Sep 29 16:34:57 dist-test-slave-mlmz krb5kdc[10430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699697, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:57.573701 10805 init.cc:377] Logged in from keytab as kudu/127.23.44.3@KRBTEST.COM (short username kudu)
I20260929 16:34:57.574463 10805 webserver.cc:460] Webserver started at http://127.23.44.3:35785/ using document root <none> and password file <none>
I20260929 16:34:57.574980 10805 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:57.575067 10805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:57.575287 10805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:57.577163 10805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "7cc155e3dba4445e8954fc099c952b82"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-mlmz"
server_key: "9884a30bae3007bd5eef48cbde2c07e2"
server_key_iv: "5421df7b351918525bd22e607e9d9685"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.577749 10805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "7cc155e3dba4445e8954fc099c952b82"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-mlmz"
server_key: "9884a30bae3007bd5eef48cbde2c07e2"
server_key_iv: "5421df7b351918525bd22e607e9d9685"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.581033 10805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:57.583328 10820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:57.584517 10805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:57.584651 10805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "7cc155e3dba4445e8954fc099c952b82"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-mlmz"
server_key: "9884a30bae3007bd5eef48cbde2c07e2"
server_key_iv: "5421df7b351918525bd22e607e9d9685"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.584753 10805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:57.598963 10805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:57.601619 10805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:57.601805 10805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:57.602321 10805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:57.603055 10805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:57.603123 10805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:57.603186 10805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:57.603233 10805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:57.611622 10805 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:38321
I20260929 16:34:57.611675 10933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:38321 every 8 connection(s)
I20260929 16:34:57.612493 10805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:34:57.617421 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 10805
I20260929 16:34:57.617549 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:34:57.617841 23728 external_mini_cluster.cc:1428] Setting key b2ae8921841a2d9774c562e1f4062dc8
Sep 29 16:34:57 dist-test-slave-mlmz krb5kdc[10430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699697, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.3@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:34:57.624861 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.614287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:59249 (local address 127.23.44.62:33025)
0929 16:34:57.614422 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:57.614431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:57.615160 (+ 729us) server_negotiation.cc:407] Connection header received
0929 16:34:57.616308 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:57.616310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:57.616355 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:57.616427 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:57.617653 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.618228 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.619361 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.619584 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.619696 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.621944 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:57.621947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:57.621949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:57.621970 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:57.623138 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:57.623598 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:57.623601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:57.623602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:57.623671 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:57.623946 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:57.623948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:57.623949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:57.624113 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:57.624185 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:57.624639 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 16:34:57.624746 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:34:57.625587 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.614602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:33025 (local address 127.23.44.3:59249)
0929 16:34:57.615008 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:57.615045 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:57.616125 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:57.616581 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:57.616587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:57.616911 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:57.617506 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.617530 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.618385 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.618388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.619237 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.619246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.619727 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.619731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.619906 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.620387 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:57.620401 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:57.621805 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:57.623239 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:57.623243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:57.623253 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:57.623515 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:57.623748 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:57.623750 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:57.623752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:57.623871 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:57.624196 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:57.624200 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:57.624409 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 16:34:57.624552 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 16:34:57.624768 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":114,"threads_started":1}
I20260929 16:34:57.626652 10934 heartbeater.cc:347] Connected to a master server at 127.23.44.62:33025
I20260929 16:34:57.626888 10934 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:57.627331 10934 heartbeater.cc:511] Master 127.23.44.62:33025 requested a full tablet report, sending...
I20260929 16:34:57.628316 10475 ts_manager.cc:194] Registered new tserver with Master: 7cc155e3dba4445e8954fc099c952b82 (127.23.44.3:38321)
I20260929 16:34:57.628885 10475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.44.3@KRBTEST.COM'} at 127.23.44.3:59249
I20260929 16:34:57.631197 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:34:57.639326 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.632938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.23.44.62:33025)
0929 16:34:57.633081 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:57.633085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:57.633174 (+ 89us) server_negotiation.cc:407] Connection header received
0929 16:34:57.633310 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:57.633317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:57.633366 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:57.633442 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:57.634065 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.634539 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.635297 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.635498 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.635570 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.636695 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:57.636698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:57.636700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:57.636720 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:57.637952 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:57.638319 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:57.638321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:57.638322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:57.638388 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:57.638603 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:57.638605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:57.638606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:57.638785 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:57.638884 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:57.639052 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 16:34:57.639131 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:34:57.642120 10475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:34:57.645005 10475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:34:57.658812 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.652611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:41271 (local address 127.0.0.1:34752)
0929 16:34:57.653540 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:57.653596 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:57.653717 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:57.654388 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:57.654396 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:57.654422 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:57.654545 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.654554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.655687 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.655689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.657209 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.657220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.658312 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.658315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.658414 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.658447 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:34:57.658557 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:34:57.658669 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":118,"thread_start_us":75,"threads_started":1}
I20260929 16:34:57.658818 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.653336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.23.44.1:41271)
0929 16:34:57.653930 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:57.653942 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:57.653973 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:34:57.654067 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:57.654072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:57.654234 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:57.654353 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:57.654685 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.655562 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.657343 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.658179 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.658312 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.658450 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:57.658586 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 16:34:57.658669 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":94,"threads_started":1}
I20260929 16:34:57.659296 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.652726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.23.44.3:38321)
0929 16:34:57.653112 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:57.653122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:57.654037 (+ 915us) server_negotiation.cc:407] Connection header received
0929 16:34:57.654145 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:57.654149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:57.654281 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:57.654450 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:57.654709 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.655830 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.657544 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.658284 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.658396 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.658522 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:57.659053 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 16:34:57.659143 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":101,"threads_started":1}
I20260929 16:34:57.659608 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.652611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:38321 (local address 127.0.0.1:42334)
0929 16:34:57.653927 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:57.653939 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:57.654035 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:57.654449 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:57.654451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:57.654469 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:57.654573 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.654580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.655955 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.655959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.657434 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.657442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.658582 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.658586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.658670 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.658693 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:34:57.658765 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 16:34:57.658830 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1255,"mutex_wait_us":284,"thread_start_us":105,"threads_started":1}
I20260929 16:34:57.659979 10945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.652859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:36177 (local address 127.0.0.1:57104)
0929 16:34:57.654590 (+ 1731us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:57.654604 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:57.654681 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:57.655162 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:57.655165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:57.655185 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:57.655266 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.655271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.656808 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.656812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.657951 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.657961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.659464 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.659467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.659637 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.659672 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:34:57.659754 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:34:57.659847 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1675,"mutex_wait_us":30,"thread_start_us":69,"threads_started":1}
I20260929 16:34:57.660337 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.653426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.23.44.2:36177)
0929 16:34:57.653852 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:57.653865 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:57.654786 (+ 921us) server_negotiation.cc:407] Connection header received
0929 16:34:57.654858 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:57.654862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:57.655019 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:57.655162 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:57.655389 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.656652 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.658091 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.659137 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.659264 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.659400 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:57.660119 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 16:34:57.660206 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":112,"threads_started":1}
I20260929 16:34:57.661870 10600 tablet_service.cc:1467] Processing CreateTablet for tablet 1b555256f92a434d9d495c72c34f365a (DEFAULT_TABLE table=test-table [id=090c94b79e9047c49047c158a622de9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:57.662703 10600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b555256f92a434d9d495c72c34f365a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:57.662331 10868 tablet_service.cc:1467] Processing CreateTablet for tablet 1b555256f92a434d9d495c72c34f365a (DEFAULT_TABLE table=test-table [id=090c94b79e9047c49047c158a622de9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:57.662737 10734 tablet_service.cc:1467] Processing CreateTablet for tablet 1b555256f92a434d9d495c72c34f365a (DEFAULT_TABLE table=test-table [id=090c94b79e9047c49047c158a622de9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:57.663277 10868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b555256f92a434d9d495c72c34f365a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:57.663630 10734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b555256f92a434d9d495c72c34f365a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:57.669064 10951 tablet_bootstrap.cc:492] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313: Bootstrap starting.
I20260929 16:34:57.669925 10952 tablet_bootstrap.cc:492] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40: Bootstrap starting.
I20260929 16:34:57.671414 10951 tablet_bootstrap.cc:654] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:57.672365 10952 tablet_bootstrap.cc:654] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:57.672801 10951 log.cc:824] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:57.673097 10952 log.cc:824] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:57.673811 10953 tablet_bootstrap.cc:492] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82: Bootstrap starting.
I20260929 16:34:57.674757 10951 tablet_bootstrap.cc:492] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313: No bootstrap required, opened a new log
I20260929 16:34:57.674957 10952 tablet_bootstrap.cc:492] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40: No bootstrap required, opened a new log
I20260929 16:34:57.674964 10951 ts_tablet_manager.cc:1397] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:34:57.675125 10952 ts_tablet_manager.cc:1397] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260929 16:34:57.675719 10953 tablet_bootstrap.cc:654] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:57.676844 10953 log.cc:824] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:57.678066 10951 raft_consensus.cc:348] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc155e3dba4445e8954fc099c952b82" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38321 } } peers { permanent_uuid: "519322c5a3634ab091de767b07dda313" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41271 } } peers { permanent_uuid: "fd9318bbde8a45b79b76decd273c2a40" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 36177 } }
I20260929 16:34:57.678292 10951 raft_consensus.cc:374] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:57.678344 10951 raft_consensus.cc:729] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519322c5a3634ab091de767b07dda313, State: Initialized, Role: FOLLOWER
I20260929 16:34:57.678756 10951 consensus_queue.cc:260] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc155e3dba4445e8954fc099c952b82" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38321 } } peers { permanent_uuid: "519322c5a3634ab091de767b07dda313" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41271 } } peers { permanent_uuid: "fd9318bbde8a45b79b76decd273c2a40" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 36177 } }
I20260929 16:34:57.678728 10952 raft_consensus.cc:348] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc155e3dba4445e8954fc099c952b82" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38321 } } peers { permanent_uuid: "519322c5a3634ab091de767b07dda313" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41271 } } peers { permanent_uuid: "fd9318bbde8a45b79b76decd273c2a40" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 36177 } }
I20260929 16:34:57.678928 10953 tablet_bootstrap.cc:492] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82: No bootstrap required, opened a new log
I20260929 16:34:57.678968 10952 raft_consensus.cc:374] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:57.679033 10952 raft_consensus.cc:729] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9318bbde8a45b79b76decd273c2a40, State: Initialized, Role: FOLLOWER
I20260929 16:34:57.679095 10953 ts_tablet_manager.cc:1397] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:34:57.679440 10952 consensus_queue.cc:260] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc155e3dba4445e8954fc099c952b82" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38321 } } peers { permanent_uuid: "519322c5a3634ab091de767b07dda313" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41271 } } peers { permanent_uuid: "fd9318bbde8a45b79b76decd273c2a40" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 36177 } }
I20260929 16:34:57.679688 10951 ts_tablet_manager.cc:1428] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:34:57.680047 10666 heartbeater.cc:503] Master 127.23.44.62:33025 was elected leader, sending a full tablet report...
I20260929 16:34:57.680342 10952 ts_tablet_manager.cc:1428] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 16:34:57.681541 10800 heartbeater.cc:503] Master 127.23.44.62:33025 was elected leader, sending a full tablet report...
I20260929 16:34:57.682662 10953 raft_consensus.cc:348] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc155e3dba4445e8954fc099c952b82" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38321 } } peers { permanent_uuid: "519322c5a3634ab091de767b07dda313" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41271 } } peers { permanent_uuid: "fd9318bbde8a45b79b76decd273c2a40" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 36177 } }
I20260929 16:34:57.682960 10953 raft_consensus.cc:374] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:57.683017 10953 raft_consensus.cc:729] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc155e3dba4445e8954fc099c952b82, State: Initialized, Role: FOLLOWER
I20260929 16:34:57.683408 10953 consensus_queue.cc:260] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc155e3dba4445e8954fc099c952b82" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38321 } } peers { permanent_uuid: "519322c5a3634ab091de767b07dda313" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41271 } } peers { permanent_uuid: "fd9318bbde8a45b79b76decd273c2a40" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 36177 } }
I20260929 16:34:57.684018 10934 heartbeater.cc:503] Master 127.23.44.62:33025 was elected leader, sending a full tablet report...
I20260929 16:34:57.684254 10953 ts_tablet_manager.cc:1428] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 16:34:57.702181 10667 tablet.cc:2369] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:34:57.735900 10959 raft_consensus.cc:482] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:34:57.736063 10959 raft_consensus.cc:504] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc155e3dba4445e8954fc099c952b82" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38321 } } peers { permanent_uuid: "519322c5a3634ab091de767b07dda313" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41271 } } peers { permanent_uuid: "fd9318bbde8a45b79b76decd273c2a40" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 36177 } }
I20260929 16:34:57.737112 10959 leader_election.cc:288] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 519322c5a3634ab091de767b07dda313 (127.23.44.1:41271), fd9318bbde8a45b79b76decd273c2a40 (127.23.44.2:36177)
I20260929 16:34:57.742025 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.737448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:39591 (local address 127.23.44.2:36177)
0929 16:34:57.737609 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:57.737613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:57.738059 (+ 446us) server_negotiation.cc:407] Connection header received
0929 16:34:57.738146 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:57.738166 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:57.738217 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:57.738328 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:57.738588 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.739208 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.740591 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.741324 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.741428 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.741505 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:57.741853 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 16:34:57.741921 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:34:57.742048 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.737587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:36177 (local address 127.23.44.3:39591)
0929 16:34:57.737967 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:57.737980 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:57.738071 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:57.738356 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:57.738358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:57.738374 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:57.738482 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.738487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.739326 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.739329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.740475 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.740482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.741569 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.741576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.741691 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.741714 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:34:57.741825 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:34:57.741896 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":30,"thread_start_us":166,"threads_started":1}
I20260929 16:34:57.742125 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.737446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:52381 (local address 127.23.44.1:41271)
0929 16:34:57.737584 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:57.737589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:57.737607 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:34:57.737758 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:57.737764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:57.737819 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:57.737907 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:57.738206 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.738961 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.740420 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.741312 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.741464 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.741552 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:57.741918 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 16:34:57.741989 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:34:57.742269 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.737329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:41271 (local address 127.23.44.3:52381)
0929 16:34:57.737532 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:57.737549 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:57.737636 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:57.737931 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:57.737934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:57.737953 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:57.738094 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.738103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.739062 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.739065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.740269 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:57.740280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.741629 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.741636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:57.741781 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.741807 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:34:57.741901 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 16:34:57.741980 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":1}
I20260929 16:34:57.742741 10754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b555256f92a434d9d495c72c34f365a" candidate_uuid: "7cc155e3dba4445e8954fc099c952b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9318bbde8a45b79b76decd273c2a40" is_pre_election: true
I20260929 16:34:57.742869 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b555256f92a434d9d495c72c34f365a" candidate_uuid: "7cc155e3dba4445e8954fc099c952b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519322c5a3634ab091de767b07dda313" is_pre_election: true
I20260929 16:34:57.743064 10754 raft_consensus.cc:2454] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cc155e3dba4445e8954fc099c952b82 in term 0.
I20260929 16:34:57.743181 10620 raft_consensus.cc:2454] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cc155e3dba4445e8954fc099c952b82 in term 0.
I20260929 16:34:57.743544 10823 leader_election.cc:302] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cc155e3dba4445e8954fc099c952b82, fd9318bbde8a45b79b76decd273c2a40; no voters:
I20260929 16:34:57.743813 10959 raft_consensus.cc:2781] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:34:57.743901 10959 raft_consensus.cc:482] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:34:57.743934 10959 raft_consensus.cc:3037] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:57.745033 10959 raft_consensus.cc:504] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc155e3dba4445e8954fc099c952b82" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38321 } } peers { permanent_uuid: "519322c5a3634ab091de767b07dda313" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41271 } } peers { permanent_uuid: "fd9318bbde8a45b79b76decd273c2a40" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 36177 } }
I20260929 16:34:57.745436 10959 leader_election.cc:288] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [CANDIDATE]: Term 1 election: Requested vote from peers 519322c5a3634ab091de767b07dda313 (127.23.44.1:41271), fd9318bbde8a45b79b76decd273c2a40 (127.23.44.2:36177)
I20260929 16:34:57.745736 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b555256f92a434d9d495c72c34f365a" candidate_uuid: "7cc155e3dba4445e8954fc099c952b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519322c5a3634ab091de767b07dda313"
I20260929 16:34:57.745862 10620 raft_consensus.cc:3037] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:57.745826 10754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b555256f92a434d9d495c72c34f365a" candidate_uuid: "7cc155e3dba4445e8954fc099c952b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9318bbde8a45b79b76decd273c2a40"
I20260929 16:34:57.746011 10754 raft_consensus.cc:3037] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:57.747102 10620 raft_consensus.cc:2454] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cc155e3dba4445e8954fc099c952b82 in term 1.
I20260929 16:34:57.747370 10754 raft_consensus.cc:2454] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cc155e3dba4445e8954fc099c952b82 in term 1.
I20260929 16:34:57.747392 10822 leader_election.cc:302] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519322c5a3634ab091de767b07dda313, 7cc155e3dba4445e8954fc099c952b82; no voters:
I20260929 16:34:57.747575 10959 raft_consensus.cc:2781] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:57.747820 10959 raft_consensus.cc:686] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 1 LEADER]: Becoming Leader. State: Replica: 7cc155e3dba4445e8954fc099c952b82, State: Running, Role: LEADER
I20260929 16:34:57.748162 10959 consensus_queue.cc:237] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc155e3dba4445e8954fc099c952b82" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38321 } } peers { permanent_uuid: "519322c5a3634ab091de767b07dda313" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41271 } } peers { permanent_uuid: "fd9318bbde8a45b79b76decd273c2a40" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 36177 } }
I20260929 16:34:57.751216 10474 catalog_manager.cc:5495] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cc155e3dba4445e8954fc099c952b82 (127.23.44.3). New cstate: current_term: 1 leader_uuid: "7cc155e3dba4445e8954fc099c952b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc155e3dba4445e8954fc099c952b82" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519322c5a3634ab091de767b07dda313" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd9318bbde8a45b79b76decd273c2a40" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 36177 } health_report { overall_health: UNKNOWN } } }
I20260929 16:34:57.778438 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.775147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.23.44.3:38321)
0929 16:34:57.775274 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:57.775278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:57.775292 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:34:57.775425 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:57.775427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:57.775477 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:57.775561 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:57.775800 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.776280 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.777073 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:57.777346 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:57.777440 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:57.777689 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:57.778079 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:57.778180 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:57.778262 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 16:34:57.778313 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:34:57.784001 10754 raft_consensus.cc:1261] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40 [term 1 FOLLOWER]: Refusing update from remote peer 7cc155e3dba4445e8954fc099c952b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:57.784060 10620 raft_consensus.cc:1261] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313 [term 1 FOLLOWER]: Refusing update from remote peer 7cc155e3dba4445e8954fc099c952b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:57.784758 10961 consensus_queue.cc:1035] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9318bbde8a45b79b76decd273c2a40" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:57.784895 10959 consensus_queue.cc:1035] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519322c5a3634ab091de767b07dda313" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 41271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:57.792137 10965 mvcc.cc:205] Tried to move back new op lower bound from 7334705962116292608 to 7334705961979170816. Current Snapshot: MvccSnapshot[applied={T|T < 7334705962116292608}]
I20260929 16:34:57.800350 10474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:57.800506 10474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:57.802893 10474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1d85b61a7e34c3a8a7c955337afe16f: Sending DeleteTablet for 3 replicas of tablet 1b555256f92a434d9d495c72c34f365a
I20260929 16:34:57.803620 10868 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b555256f92a434d9d495c72c34f365a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:57 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:42334
I20260929 16:34:57.803680 10600 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b555256f92a434d9d495c72c34f365a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:57 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:34752
I20260929 16:34:57.803769 10734 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b555256f92a434d9d495c72c34f365a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:57 UTC) from {username='kudu', principal='kudu/127.23.44.62@KRBTEST.COM'} at 127.0.0.1:57104
I20260929 16:34:57.804155 10973 tablet_replica.cc:330] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82: stopping tablet replica
I20260929 16:34:57.804436 10974 tablet_replica.cc:330] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313: stopping tablet replica
I20260929 16:34:57.804468 10973 raft_consensus.cc:2229] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:34:57.804574 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 10537
I20260929 16:34:57.804790 10974 raft_consensus.cc:2229] T 1b555256f92a434d9d495c72c34f365a P 519322c5a3634ab091de767b07dda313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:34:57.805235 10975 tablet_replica.cc:330] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40: stopping tablet replica
I20260929 16:34:57.805540 10975 raft_consensus.cc:2229] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:34:57.805718 10973 raft_consensus.cc:2258] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:57.805861 10975 raft_consensus.cc:2258] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:57.807250 10975 ts_tablet_manager.cc:1905] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:57.807256 10973 ts_tablet_manager.cc:1905] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:57.809933 10975 ts_tablet_manager.cc:1918] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:57.810082 10975 log.cc:1196] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/1b555256f92a434d9d495c72c34f365a
I20260929 16:34:57.810477 10975 ts_tablet_manager.cc:1939] T 1b555256f92a434d9d495c72c34f365a P fd9318bbde8a45b79b76decd273c2a40: Deleting consensus metadata
I20260929 16:34:57.810781 10973 ts_tablet_manager.cc:1918] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:57.810886 10973 log.cc:1196] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/1b555256f92a434d9d495c72c34f365a
I20260929 16:34:57.811241 10973 ts_tablet_manager.cc:1939] T 1b555256f92a434d9d495c72c34f365a P 7cc155e3dba4445e8954fc099c952b82: Deleting consensus metadata
I20260929 16:34:57.811899 10461 catalog_manager.cc:4841] TS fd9318bbde8a45b79b76decd273c2a40 (127.23.44.2:36177): tablet 1b555256f92a434d9d495c72c34f365a (table test-table [id=090c94b79e9047c49047c158a622de9e]) successfully deleted
I20260929 16:34:57.812459 10459 catalog_manager.cc:4841] TS 7cc155e3dba4445e8954fc099c952b82 (127.23.44.3:38321): tablet 1b555256f92a434d9d495c72c34f365a (table test-table [id=090c94b79e9047c49047c158a622de9e]) successfully deleted
W20260929 16:34:57.814028 10460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:41271 (error 108)
I20260929 16:34:57.814291 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 10671
I20260929 16:34:57.815940 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:57.815586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:41271 (local address 127.0.0.1:34766)
0929 16:34:57.815751 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:57.815834 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:34:57.816079 10460 catalog_manager.cc:4584] TS 519322c5a3634ab091de767b07dda313 (127.23.44.1:41271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b555256f92a434d9d495c72c34f365a: Network error: Client connection negotiation failed: client connection to 127.23.44.1:41271: connect: Connection refused (error 111)
I20260929 16:34:57.820390 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 10805
I20260929 16:34:57.826359 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 10443
2026-09-29T16:34:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2214 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T16:34:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:57Z Disabled control of system clock
I20260929 16:34:57.856030 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:34567
--webserver_interface=127.23.44.62
--webserver_port=0
--builtin_ntp_servers=127.23.44.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:34567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:34:57.961853 10980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:57.962081 10980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:57.962114 10980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:57.965962 10980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:57.966007 10980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:57.966022 10980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:57.966039 10980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:57.966053 10980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:57.970662 10980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:34567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:34567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:57.971572 10980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:57.972815 10980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:34:57.981092 10980 server_base.cc:1034] running on GCE node
W20260929 16:34:57.981011 10986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:57.981050 10988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:57.981006 10985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:57.981810 10980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:57.982836 10980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:57.984030 10980 hybrid_clock.cc:648] HybridClock initialized: now 1790699697984019 us; error 26 us; skew 500 ppm
I20260929 16:34:57.985956 10980 webserver.cc:460] Webserver started at http://127.23.44.62:33419/ using document root <none> and password file <none>
I20260929 16:34:57.986533 10980 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:57.986629 10980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:57.986850 10980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:57.988852 10980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "ba5ff1d6ae0b4c26b5b56bb142bd9e4a"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-mlmz"
server_key: "6351a41bcf2aa1785f614f32bdc3cf55"
server_key_iv: "01ba7b14c9189c021e1c3b9bb0ac2cd5"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.989468 10980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "ba5ff1d6ae0b4c26b5b56bb142bd9e4a"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-mlmz"
server_key: "6351a41bcf2aa1785f614f32bdc3cf55"
server_key_iv: "01ba7b14c9189c021e1c3b9bb0ac2cd5"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.992942 10980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:34:57.995307 10994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:57.996407 10980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:34:57.996541 10980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "ba5ff1d6ae0b4c26b5b56bb142bd9e4a"
format_stamp: "Formatted at 2026-09-29 16:34:57 on dist-test-slave-mlmz"
server_key: "6351a41bcf2aa1785f614f32bdc3cf55"
server_key_iv: "01ba7b14c9189c021e1c3b9bb0ac2cd5"
server_key_version: "encryptionkey@0"
I20260929 16:34:57.996646 10980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:58.028623 10980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:58.029345 10980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:58.029528 10980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:58.036300 10980 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:34567
I20260929 16:34:58.036309 11046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:34567 every 8 connection(s)
I20260929 16:34:58.037163 10980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:58.039971 11047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:58.041474 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 10980
I20260929 16:34:58.041617 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:34:58.041898 23728 external_mini_cluster.cc:1428] Setting key 497b8e31e5008b52754b651897e9e57f
I20260929 16:34:58.046527 11047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a: Bootstrap starting.
I20260929 16:34:58.048869 11047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:58.049659 11047 log.cc:824] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:58.051667 11047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a: No bootstrap required, opened a new log
I20260929 16:34:58.051714 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.043225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54466 (local address 127.23.44.62:34567)
0929 16:34:58.043855 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.043875 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.043916 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:34:58.044690 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.044711 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.045167 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.045523 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:58.045938 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.047341 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.048210 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.048870 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.049030 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.049887 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:58.049892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:58.049907 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:58.050308 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:58.050392 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:58.050419 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:58.050518 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.050661 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.050895 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260929 16:34:58.055250 11047 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5ff1d6ae0b4c26b5b56bb142bd9e4a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34567 } }
I20260929 16:34:58.055445 11047 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:58.055480 11047 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5ff1d6ae0b4c26b5b56bb142bd9e4a, State: Initialized, Role: FOLLOWER
I20260929 16:34:58.055799 11047 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5ff1d6ae0b4c26b5b56bb142bd9e4a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34567 } }
I20260929 16:34:58.055896 11047 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:58.055939 11047 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:58.056003 11047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:58.056717 11047 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5ff1d6ae0b4c26b5b56bb142bd9e4a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34567 } }
I20260929 16:34:58.056972 11047 leader_election.cc:302] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba5ff1d6ae0b4c26b5b56bb142bd9e4a; no voters:
I20260929 16:34:58.057327 11047 leader_election.cc:288] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:58.057384 11052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:58.057585 11052 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [term 1 LEADER]: Becoming Leader. State: Replica: ba5ff1d6ae0b4c26b5b56bb142bd9e4a, State: Running, Role: LEADER
I20260929 16:34:58.057865 11052 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5ff1d6ae0b4c26b5b56bb142bd9e4a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34567 } }
I20260929 16:34:58.058476 11047 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:58.059429 11053 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba5ff1d6ae0b4c26b5b56bb142bd9e4a. Latest consensus state: current_term: 1 leader_uuid: "ba5ff1d6ae0b4c26b5b56bb142bd9e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5ff1d6ae0b4c26b5b56bb142bd9e4a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34567 } } }
I20260929 16:34:58.059552 11053 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:58.059806 11054 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba5ff1d6ae0b4c26b5b56bb142bd9e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5ff1d6ae0b4c26b5b56bb142bd9e4a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 34567 } } }
I20260929 16:34:58.059896 11054 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:58.059937 11061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:58.062851 11061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:58.067415 11061 catalog_manager.cc:1347] Generated new cluster ID: ec65844a4c094073897cc8128b479dea
I20260929 16:34:58.067473 11061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:58.075064 11061 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:58.075789 11061 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:58.079841 11061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a: Generated new TSK 0
I20260929 16:34:58.080405 11061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:58.085827 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.1:0
--local_ip_for_outbound_sockets=127.23.44.1
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34567
--builtin_ntp_servers=127.23.44.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:34:58.209034 11071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:58.209370 11071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:58.209437 11071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:58.213949 11071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:58.213994 11071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:58.214066 11071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.1
I20260929 16:34:58.218806 11071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.44.1
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.1
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:58.219732 11071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:58.220983 11071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:58.229686 11079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:58.229705 11076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:58.229698 11077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:58.229877 11071 server_base.cc:1034] running on GCE node
I20260929 16:34:58.230293 11071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:58.230849 11071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:58.232019 11071 hybrid_clock.cc:648] HybridClock initialized: now 1790699698231995 us; error 36 us; skew 500 ppm
I20260929 16:34:58.233734 11071 webserver.cc:460] Webserver started at http://127.23.44.1:41701/ using document root <none> and password file <none>
I20260929 16:34:58.234252 11071 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:58.234309 11071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:58.234547 11071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:58.236392 11071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/data/instance:
uuid: "cdf6fade847a448f80a2fd5f200a82c6"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-mlmz"
server_key: "d92783a679bb6f431be06d30ff7350cd"
server_key_iv: "43bdbbcdc29c03bb4bec72d63813c4ee"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.236922 11071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance:
uuid: "cdf6fade847a448f80a2fd5f200a82c6"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-mlmz"
server_key: "d92783a679bb6f431be06d30ff7350cd"
server_key_iv: "43bdbbcdc29c03bb4bec72d63813c4ee"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.240217 11071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:34:58.242470 11085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.243655 11071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:34:58.243788 11071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/wal
uuid: "cdf6fade847a448f80a2fd5f200a82c6"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-mlmz"
server_key: "d92783a679bb6f431be06d30ff7350cd"
server_key_iv: "43bdbbcdc29c03bb4bec72d63813c4ee"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.243885 11071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:58.259690 11071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:58.260313 11071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:58.260493 11071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:58.261041 11071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:58.261955 11071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:58.262009 11071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.262072 11071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:58.262104 11071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.270565 11071 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.1:40511
I20260929 16:34:58.270603 11198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.1:40511 every 8 connection(s)
I20260929 16:34:58.271371 11071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/data/info.pb
I20260929 16:34:58.271997 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 11071
I20260929 16:34:58.272125 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-0/wal/instance
I20260929 16:34:58.272410 23728 external_mini_cluster.cc:1428] Setting key f30da98c5391456931ca471ad5597ae7
I20260929 16:34:58.274363 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.2:0
--local_ip_for_outbound_sockets=127.23.44.2
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34567
--builtin_ntp_servers=127.23.44.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:34:58.281626 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.273217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.1:57331 (local address 127.23.44.62:34567)
0929 16:34:58.273375 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.273379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.274502 (+ 1123us) server_negotiation.cc:407] Connection header received
0929 16:34:58.276084 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.276086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.276128 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.276198 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:58.277142 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.277685 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.279460 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.279667 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.279750 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.281096 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:58.281100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:58.281103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:58.281115 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:58.281136 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:58.281145 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:58.281238 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.281460 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.281523 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:34:58.283100 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.273650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:34567 (local address 127.23.44.1:57331)
0929 16:34:58.274368 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:58.274414 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:58.275855 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:58.276370 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:58.276380 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:58.276499 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:58.277008 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.277021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.277822 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.277829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.279338 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.279352 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.279830 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.279836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.280068 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.280753 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:34:58.280757 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:58.280816 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:34:58.280906 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:58.281253 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:58.281288 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:58.281296 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:34:58.281437 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 16:34:58.281748 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":135,"threads_started":1}
I20260929 16:34:58.284337 11199 heartbeater.cc:347] Connected to a master server at 127.23.44.62:34567
I20260929 16:34:58.284549 11199 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:58.284998 11199 heartbeater.cc:511] Master 127.23.44.62:34567 requested a full tablet report, sending...
I20260929 16:34:58.286477 11011 ts_manager.cc:194] Registered new tserver with Master: cdf6fade847a448f80a2fd5f200a82c6 (127.23.44.1:40511)
I20260929 16:34:58.287536 11011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.44.1:57331
W20260929 16:34:58.381407 11202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:58.381635 11202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:58.381671 11202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:58.385710 11202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:58.385756 11202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:58.385840 11202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.2
I20260929 16:34:58.390564 11202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.44.2
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.2
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:58.391518 11202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:58.392800 11202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:58.402230 11208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:58.402297 11207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:58.402493 11202 server_base.cc:1034] running on GCE node
W20260929 16:34:58.402247 11210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:58.403127 11202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:58.403779 11202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:58.404980 11202 hybrid_clock.cc:648] HybridClock initialized: now 1790699698404950 us; error 41 us; skew 500 ppm
I20260929 16:34:58.406889 11202 webserver.cc:460] Webserver started at http://127.23.44.2:38227/ using document root <none> and password file <none>
I20260929 16:34:58.407455 11202 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:58.407550 11202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:58.407775 11202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:58.409829 11202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/data/instance:
uuid: "f8297aa01f3e47339fe1a0227c002706"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-mlmz"
server_key: "061e3cd0d73ef9b65ff3f0e6cf7fad43"
server_key_iv: "95498b3f59914f53149df9d2a88136bc"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.410379 11202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance:
uuid: "f8297aa01f3e47339fe1a0227c002706"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-mlmz"
server_key: "061e3cd0d73ef9b65ff3f0e6cf7fad43"
server_key_iv: "95498b3f59914f53149df9d2a88136bc"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.413823 11202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:58.416272 11216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.417434 11202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:34:58.417557 11202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/wal
uuid: "f8297aa01f3e47339fe1a0227c002706"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-mlmz"
server_key: "061e3cd0d73ef9b65ff3f0e6cf7fad43"
server_key_iv: "95498b3f59914f53149df9d2a88136bc"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.417665 11202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:58.442620 11202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:58.443321 11202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:58.443524 11202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:58.444121 11202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:58.445066 11202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:58.445147 11202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.445221 11202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:58.445274 11202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.454216 11202 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.2:35695
I20260929 16:34:58.454319 11329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.2:35695 every 8 connection(s)
I20260929 16:34:58.455132 11202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/data/info.pb
I20260929 16:34:58.461171 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 11202
I20260929 16:34:58.461323 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/wal/instance
I20260929 16:34:58.461647 23728 external_mini_cluster.cc:1428] Setting key 2c3416fafd14d39c75d9dacce5558769
I20260929 16:34:58.463649 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.44.3:0
--local_ip_for_outbound_sockets=127.23.44.3
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34567
--builtin_ntp_servers=127.23.44.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:34:58.465099 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.456956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:55759 (local address 127.23.44.62:34567)
0929 16:34:58.457115 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.457119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.458170 (+ 1051us) server_negotiation.cc:407] Connection header received
0929 16:34:58.459768 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.459771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.459818 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.459895 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:58.460823 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.461659 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.462945 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.463226 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.463335 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.464527 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:58.464533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:58.464539 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:58.464557 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:58.464584 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:58.464597 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:58.464688 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.464824 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.464897 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:34:58.466190 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.457368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:34567 (local address 127.23.44.2:55759)
0929 16:34:58.458017 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:58.458055 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:58.459571 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:58.460091 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:58.460097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:58.460185 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:58.460698 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.460710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.461808 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.461811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.462798 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.462808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.463382 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.463384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.463527 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.464277 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:34:58.464278 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:58.464316 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:34:58.464376 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:58.464692 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:58.464698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:58.464702 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:34:58.464792 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:34:58.465101 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
I20260929 16:34:58.467310 11330 heartbeater.cc:347] Connected to a master server at 127.23.44.62:34567
I20260929 16:34:58.467540 11330 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:58.468041 11330 heartbeater.cc:511] Master 127.23.44.62:34567 requested a full tablet report, sending...
I20260929 16:34:58.469079 11011 ts_manager.cc:194] Registered new tserver with Master: f8297aa01f3e47339fe1a0227c002706 (127.23.44.2:35695)
I20260929 16:34:58.469688 11011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.44.2:55759
W20260929 16:34:58.580114 11333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:58.580335 11333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:58.580400 11333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:58.584342 11333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:58.584393 11333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:58.584465 11333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.44.3
I20260929 16:34:58.589066 11333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.44.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.44.3
--webserver_port=0
--tserver_master_addrs=127.23.44.62:34567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.44.3
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:34:58.590024 11333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:58.591219 11333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:58.600260 11341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:58.600252 11339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:58.600252 11338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:58.600586 11333 server_base.cc:1034] running on GCE node
I20260929 16:34:58.600975 11333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:58.601610 11333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:58.602818 11333 hybrid_clock.cc:648] HybridClock initialized: now 1790699698602788 us; error 45 us; skew 500 ppm
I20260929 16:34:58.604588 11333 webserver.cc:460] Webserver started at http://127.23.44.3:38105/ using document root <none> and password file <none>
I20260929 16:34:58.605122 11333 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:58.605212 11333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:58.605444 11333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:58.607334 11333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/data/instance:
uuid: "814f2149ac9a46ec88b37df6dee6741f"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-mlmz"
server_key: "333dfd7022c10fa27a1da4af467e114c"
server_key_iv: "9a1d89870788f0676d4a5b0d9754cd68"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.607868 11333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance:
uuid: "814f2149ac9a46ec88b37df6dee6741f"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-mlmz"
server_key: "333dfd7022c10fa27a1da4af467e114c"
server_key_iv: "9a1d89870788f0676d4a5b0d9754cd68"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.611307 11333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:34:58.613659 11347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.615003 11333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:34:58.615109 11333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/wal
uuid: "814f2149ac9a46ec88b37df6dee6741f"
format_stamp: "Formatted at 2026-09-29 16:34:58 on dist-test-slave-mlmz"
server_key: "333dfd7022c10fa27a1da4af467e114c"
server_key_iv: "9a1d89870788f0676d4a5b0d9754cd68"
server_key_version: "encryptionkey@0"
I20260929 16:34:58.615227 11333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:58.643186 11333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:58.643781 11333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:58.643919 11333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:58.644428 11333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:34:58.645278 11333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:34:58.645323 11333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.645360 11333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:34:58.645375 11333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:58.653983 11333 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.3:37597
I20260929 16:34:58.654019 11460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.3:37597 every 8 connection(s)
I20260929 16:34:58.654807 11333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/data/info.pb
I20260929 16:34:58.660331 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 11333
I20260929 16:34:58.660463 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/wal/instance
I20260929 16:34:58.660792 23728 external_mini_cluster.cc:1428] Setting key 1917d75a08eb258850378e856c543b66
I20260929 16:34:58.663977 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.656466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.3:57193 (local address 127.23.44.62:34567)
0929 16:34:58.656597 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.656601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.657566 (+ 965us) server_negotiation.cc:407] Connection header received
0929 16:34:58.658945 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.658948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.659005 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.659083 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:58.660006 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.660641 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.662078 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.662360 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.662445 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.663525 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:58.663528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:58.663530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:58.663540 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:58.663558 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:58.663566 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:58.663634 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.663825 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.663875 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:34:58.664718 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.656797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:34567 (local address 127.23.44.3:57193)
0929 16:34:58.657430 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:58.657467 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:58.658688 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:58.659235 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:58.659241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:58.659319 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:58.659864 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.659883 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.660767 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.660771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.661945 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.661957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.662510 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.662514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.662700 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.663281 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:34:58.663283 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:58.663319 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:34:58.663381 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:58.663645 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:58.663650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:58.663654 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:34:58.663741 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:34:58.663942 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
I20260929 16:34:58.665781 11461 heartbeater.cc:347] Connected to a master server at 127.23.44.62:34567
I20260929 16:34:58.665992 11461 heartbeater.cc:464] Registering TS with master...
I20260929 16:34:58.666445 11461 heartbeater.cc:511] Master 127.23.44.62:34567 requested a full tablet report, sending...
I20260929 16:34:58.667400 11011 ts_manager.cc:194] Registered new tserver with Master: 814f2149ac9a46ec88b37df6dee6741f (127.23.44.3:37597)
I20260929 16:34:58.667924 11011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.44.3:57193
I20260929 16:34:58.674480 23728 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:34:58.679767 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.676320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.23.44.62:34567)
0929 16:34:58.676446 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.676450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.676580 (+ 130us) server_negotiation.cc:407] Connection header received
0929 16:34:58.676671 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.676673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.676707 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.676769 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:58.677034 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.677620 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.678421 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.678621 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.678709 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.679395 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:58.679397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:58.679399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:58.679410 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:58.679426 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:58.679433 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:58.679490 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.679559 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.679609 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:34:58.682437 11011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:34:58.685319 11011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:34:58.698592 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.693537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.23.44.3:37597)
0929 16:34:58.694325 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.694339 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.694362 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:34:58.694427 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.694430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.694555 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.694699 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:58.694948 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.695641 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.697188 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.697888 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.698013 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.698207 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.698310 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.698407 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":120,"threads_started":1}
I20260929 16:34:58.698614 11472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.692782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.2:35695 (local address 127.0.0.1:44314)
0929 16:34:58.693520 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:58.693568 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:58.693755 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:58.694379 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:58.694386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:58.694410 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:58.694533 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.694540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.696096 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.696099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.697229 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.697242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.698198 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.698202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.698293 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.698318 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:34:58.698369 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 16:34:58.698443 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":32,"thread_start_us":78,"threads_started":1}
I20260929 16:34:58.699218 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.693499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:40511 (local address 127.0.0.1:53556)
0929 16:34:58.693994 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:58.694006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:58.694080 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:58.694759 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:58.694762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:58.694780 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:58.694878 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.694884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.696299 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.696305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.697509 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.697520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.698531 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.698534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.698627 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.698960 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 16:34:58.699060 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:34:58.699126 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":7,"thread_start_us":96,"threads_started":1}
I20260929 16:34:58.699368 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.692899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.23.44.2:35695)
0929 16:34:58.693336 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.693350 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.693649 (+ 299us) server_negotiation.cc:407] Connection header received
0929 16:34:58.693975 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.693982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.694194 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.694688 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:58.694839 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.695976 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.697373 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.698060 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.698920 (+ 860us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.699067 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.699114 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.699204 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":122,"threads_started":1}
I20260929 16:34:58.699378 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.693931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.23.44.1:40511)
0929 16:34:58.694324 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.694337 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.694360 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:34:58.694427 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.694431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.694613 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.695023 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:58.695156 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.696188 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.697639 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.698435 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.698530 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.698698 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.699116 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.699205 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":88,"threads_started":1}
I20260929 16:34:58.699568 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.692873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:37597 (local address 127.0.0.1:54414)
0929 16:34:58.693688 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:58.693701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:58.693782 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:58.694692 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:58.694695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:58.694712 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:58.694826 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.694833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.695758 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.695762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.697060 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.697070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.698011 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.698015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.698121 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.698157 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:34:58.698285 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 16:34:58.698408 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":93,"threads_started":1}
I20260929 16:34:58.702036 11133 tablet_service.cc:1467] Processing CreateTablet for tablet 3362652a3c274eecb35285d3d453560a (DEFAULT_TABLE table=test-table [id=2e6f2eea05be474581722ebcbdc262c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:58.702126 11264 tablet_service.cc:1467] Processing CreateTablet for tablet 3362652a3c274eecb35285d3d453560a (DEFAULT_TABLE table=test-table [id=2e6f2eea05be474581722ebcbdc262c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:58.702128 11395 tablet_service.cc:1467] Processing CreateTablet for tablet 3362652a3c274eecb35285d3d453560a (DEFAULT_TABLE table=test-table [id=2e6f2eea05be474581722ebcbdc262c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:34:58.703051 11133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3362652a3c274eecb35285d3d453560a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:58.703074 11395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3362652a3c274eecb35285d3d453560a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:58.703074 11264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3362652a3c274eecb35285d3d453560a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:58.709589 11478 tablet_bootstrap.cc:492] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6: Bootstrap starting.
I20260929 16:34:58.710250 11480 tablet_bootstrap.cc:492] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f: Bootstrap starting.
I20260929 16:34:58.711589 11478 tablet_bootstrap.cc:654] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:58.711855 11480 tablet_bootstrap.cc:654] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:58.712366 11478 log.cc:824] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:58.712512 11480 log.cc:824] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:58.713080 11479 tablet_bootstrap.cc:492] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706: Bootstrap starting.
I20260929 16:34:58.714350 11478 tablet_bootstrap.cc:492] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6: No bootstrap required, opened a new log
I20260929 16:34:58.714350 11480 tablet_bootstrap.cc:492] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f: No bootstrap required, opened a new log
I20260929 16:34:58.714540 11478 ts_tablet_manager.cc:1397] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:34:58.714552 11480 ts_tablet_manager.cc:1397] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 16:34:58.714870 11479 tablet_bootstrap.cc:654] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:58.715552 11479 log.cc:824] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:58.717212 11479 tablet_bootstrap.cc:492] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706: No bootstrap required, opened a new log
I20260929 16:34:58.717439 11479 ts_tablet_manager.cc:1397] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:34:58.718888 11478 raft_consensus.cc:348] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8297aa01f3e47339fe1a0227c002706" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35695 } } peers { permanent_uuid: "814f2149ac9a46ec88b37df6dee6741f" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 37597 } } peers { permanent_uuid: "cdf6fade847a448f80a2fd5f200a82c6" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 40511 } }
I20260929 16:34:58.718888 11480 raft_consensus.cc:348] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8297aa01f3e47339fe1a0227c002706" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35695 } } peers { permanent_uuid: "814f2149ac9a46ec88b37df6dee6741f" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 37597 } } peers { permanent_uuid: "cdf6fade847a448f80a2fd5f200a82c6" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 40511 } }
I20260929 16:34:58.719139 11480 raft_consensus.cc:374] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:58.719139 11478 raft_consensus.cc:374] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:58.719195 11480 raft_consensus.cc:729] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814f2149ac9a46ec88b37df6dee6741f, State: Initialized, Role: FOLLOWER
I20260929 16:34:58.719195 11478 raft_consensus.cc:729] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdf6fade847a448f80a2fd5f200a82c6, State: Initialized, Role: FOLLOWER
I20260929 16:34:58.719628 11480 consensus_queue.cc:260] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8297aa01f3e47339fe1a0227c002706" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35695 } } peers { permanent_uuid: "814f2149ac9a46ec88b37df6dee6741f" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 37597 } } peers { permanent_uuid: "cdf6fade847a448f80a2fd5f200a82c6" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 40511 } }
I20260929 16:34:58.719628 11478 consensus_queue.cc:260] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8297aa01f3e47339fe1a0227c002706" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35695 } } peers { permanent_uuid: "814f2149ac9a46ec88b37df6dee6741f" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 37597 } } peers { permanent_uuid: "cdf6fade847a448f80a2fd5f200a82c6" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 40511 } }
I20260929 16:34:58.720248 11199 heartbeater.cc:503] Master 127.23.44.62:34567 was elected leader, sending a full tablet report...
I20260929 16:34:58.720579 11480 ts_tablet_manager.cc:1428] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:34:58.721303 11461 heartbeater.cc:503] Master 127.23.44.62:34567 was elected leader, sending a full tablet report...
I20260929 16:34:58.722482 11478 ts_tablet_manager.cc:1428] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 16:34:58.721189 11479 raft_consensus.cc:348] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8297aa01f3e47339fe1a0227c002706" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35695 } } peers { permanent_uuid: "814f2149ac9a46ec88b37df6dee6741f" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 37597 } } peers { permanent_uuid: "cdf6fade847a448f80a2fd5f200a82c6" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 40511 } }
I20260929 16:34:58.722970 11479 raft_consensus.cc:374] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:58.723035 11479 raft_consensus.cc:729] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8297aa01f3e47339fe1a0227c002706, State: Initialized, Role: FOLLOWER
I20260929 16:34:58.723450 11479 consensus_queue.cc:260] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8297aa01f3e47339fe1a0227c002706" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35695 } } peers { permanent_uuid: "814f2149ac9a46ec88b37df6dee6741f" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 37597 } } peers { permanent_uuid: "cdf6fade847a448f80a2fd5f200a82c6" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 40511 } }
I20260929 16:34:58.724038 11330 heartbeater.cc:503] Master 127.23.44.62:34567 was elected leader, sending a full tablet report...
I20260929 16:34:58.724395 11479 ts_tablet_manager.cc:1428] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260929 16:34:58.772558 11200 tablet.cc:2369] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:34:58.905711 11462 tablet.cc:2369] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:34:58.948220 11486 raft_consensus.cc:482] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:34:58.948467 11486 raft_consensus.cc:504] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8297aa01f3e47339fe1a0227c002706" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35695 } } peers { permanent_uuid: "814f2149ac9a46ec88b37df6dee6741f" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 37597 } } peers { permanent_uuid: "cdf6fade847a448f80a2fd5f200a82c6" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 40511 } }
I20260929 16:34:58.949615 11486 leader_election.cc:288] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 814f2149ac9a46ec88b37df6dee6741f (127.23.44.3:37597), cdf6fade847a448f80a2fd5f200a82c6 (127.23.44.1:40511)
I20260929 16:34:58.953871 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.949891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.3:37597 (local address 127.23.44.2:52377)
0929 16:34:58.949997 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:58.950012 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:58.950105 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:58.950494 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:58.950497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:58.950518 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:58.950639 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.950646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.951519 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.951522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.952516 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.952525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.953441 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.953445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.953544 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.953569 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:34:58.953651 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:34:58.953720 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 16:34:58.953860 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.950002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:52377 (local address 127.23.44.3:37597)
0929 16:34:58.950121 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.950126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.950144 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:34:58.950309 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.950316 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.950377 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.950457 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:58.950753 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.951342 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.952649 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.953306 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.953399 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.953448 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.953700 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.953763 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:34:58.954479 11487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.949891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:40511 (local address 127.23.44.2:39801)
0929 16:34:58.950275 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:58.950291 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:58.950391 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:58.950648 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:58.950650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:58.950664 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:34:58.950819 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.950826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.952070 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.952072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.953063 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:58.953074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.954103 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.954109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:58.954205 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.954232 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:34:58.954326 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 16:34:58.954379 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":114,"thread_start_us":67,"threads_started":1}
I20260929 16:34:58.954542 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.950002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.44.2:39801 (local address 127.23.44.1:40511)
0929 16:34:58.950121 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.950126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.950384 (+ 258us) server_negotiation.cc:407] Connection header received
0929 16:34:58.950497 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.950500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.950549 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.950636 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:34:58.951075 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.951930 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.953206 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.953814 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.953925 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.953985 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.954356 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.954433 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:34:58.954591 11415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3362652a3c274eecb35285d3d453560a" candidate_uuid: "f8297aa01f3e47339fe1a0227c002706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "814f2149ac9a46ec88b37df6dee6741f" is_pre_election: true
I20260929 16:34:58.954941 11415 raft_consensus.cc:2454] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8297aa01f3e47339fe1a0227c002706 in term 0.
I20260929 16:34:58.955219 11153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3362652a3c274eecb35285d3d453560a" candidate_uuid: "f8297aa01f3e47339fe1a0227c002706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdf6fade847a448f80a2fd5f200a82c6" is_pre_election: true
I20260929 16:34:58.955487 11153 raft_consensus.cc:2454] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8297aa01f3e47339fe1a0227c002706 in term 0.
I20260929 16:34:58.955482 11217 leader_election.cc:302] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 814f2149ac9a46ec88b37df6dee6741f, f8297aa01f3e47339fe1a0227c002706; no voters:
I20260929 16:34:58.955729 11486 raft_consensus.cc:2781] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:34:58.955813 11486 raft_consensus.cc:482] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:34:58.955865 11486 raft_consensus.cc:3037] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 0 FOLLOWER]: Advancing to term 1
W20260929 16:34:58.956442 11331 tablet.cc:2369] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:34:58.956930 11486 raft_consensus.cc:504] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8297aa01f3e47339fe1a0227c002706" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35695 } } peers { permanent_uuid: "814f2149ac9a46ec88b37df6dee6741f" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 37597 } } peers { permanent_uuid: "cdf6fade847a448f80a2fd5f200a82c6" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 40511 } }
I20260929 16:34:58.957329 11486 leader_election.cc:288] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [CANDIDATE]: Term 1 election: Requested vote from peers 814f2149ac9a46ec88b37df6dee6741f (127.23.44.3:37597), cdf6fade847a448f80a2fd5f200a82c6 (127.23.44.1:40511)
I20260929 16:34:58.957608 11415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3362652a3c274eecb35285d3d453560a" candidate_uuid: "f8297aa01f3e47339fe1a0227c002706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "814f2149ac9a46ec88b37df6dee6741f"
I20260929 16:34:58.957652 11153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3362652a3c274eecb35285d3d453560a" candidate_uuid: "f8297aa01f3e47339fe1a0227c002706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdf6fade847a448f80a2fd5f200a82c6"
I20260929 16:34:58.957751 11153 raft_consensus.cc:3037] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:58.957733 11415 raft_consensus.cc:3037] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:58.958993 11153 raft_consensus.cc:2454] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8297aa01f3e47339fe1a0227c002706 in term 1.
I20260929 16:34:58.958993 11415 raft_consensus.cc:2454] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8297aa01f3e47339fe1a0227c002706 in term 1.
I20260929 16:34:58.959268 11217 leader_election.cc:302] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 814f2149ac9a46ec88b37df6dee6741f, f8297aa01f3e47339fe1a0227c002706; no voters:
I20260929 16:34:58.959492 11486 raft_consensus.cc:2781] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:58.959764 11486 raft_consensus.cc:686] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 1 LEADER]: Becoming Leader. State: Replica: f8297aa01f3e47339fe1a0227c002706, State: Running, Role: LEADER
I20260929 16:34:58.960081 11486 consensus_queue.cc:237] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8297aa01f3e47339fe1a0227c002706" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35695 } } peers { permanent_uuid: "814f2149ac9a46ec88b37df6dee6741f" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 37597 } } peers { permanent_uuid: "cdf6fade847a448f80a2fd5f200a82c6" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 40511 } }
I20260929 16:34:58.963037 11011 catalog_manager.cc:5495] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8297aa01f3e47339fe1a0227c002706 (127.23.44.2). New cstate: current_term: 1 leader_uuid: "f8297aa01f3e47339fe1a0227c002706" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8297aa01f3e47339fe1a0227c002706" member_type: VOTER last_known_addr { host: "127.23.44.2" port: 35695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "814f2149ac9a46ec88b37df6dee6741f" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 37597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdf6fade847a448f80a2fd5f200a82c6" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 40511 } health_report { overall_health: UNKNOWN } } }
I20260929 16:34:58.986814 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:58.983374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44330 (local address 127.23.44.2:35695)
0929 16:34:58.983520 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:58.983525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:58.983540 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:34:58.983632 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:58.983634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:58.983685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:58.983797 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:58.984043 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.984547 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.985425 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:58.985637 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:58.985748 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:58.986041 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:58.986432 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:58.986559 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:58.986626 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 16:34:58.986676 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:34:58.992530 11415 raft_consensus.cc:1261] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f [term 1 FOLLOWER]: Refusing update from remote peer f8297aa01f3e47339fe1a0227c002706: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:58.992530 11153 raft_consensus.cc:1261] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6 [term 1 FOLLOWER]: Refusing update from remote peer f8297aa01f3e47339fe1a0227c002706: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:58.993290 11486 consensus_queue.cc:1035] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "814f2149ac9a46ec88b37df6dee6741f" member_type: VOTER last_known_addr { host: "127.23.44.3" port: 37597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:58.993513 11488 consensus_queue.cc:1035] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdf6fade847a448f80a2fd5f200a82c6" member_type: VOTER last_known_addr { host: "127.23.44.1" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:59.009507 11010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:59.009678 11010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:34:59.011958 11010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba5ff1d6ae0b4c26b5b56bb142bd9e4a: Sending DeleteTablet for 3 replicas of tablet 3362652a3c274eecb35285d3d453560a
I20260929 16:34:59.012684 11264 tablet_service.cc:1514] Processing DeleteTablet for tablet 3362652a3c274eecb35285d3d453560a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:59 UTC) from {username='slave'} at 127.0.0.1:44314
I20260929 16:34:59.013088 11133 tablet_service.cc:1514] Processing DeleteTablet for tablet 3362652a3c274eecb35285d3d453560a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:59 UTC) from {username='slave'} at 127.0.0.1:53556
I20260929 16:34:59.013149 11395 tablet_service.cc:1514] Processing DeleteTablet for tablet 3362652a3c274eecb35285d3d453560a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:34:59 UTC) from {username='slave'} at 127.0.0.1:54414
I20260929 16:34:59.013517 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 11071
I20260929 16:34:59.013522 11501 tablet_replica.cc:330] T 3362652a3c274eecb35285d3d453560a P cdf6fade847a448f80a2fd5f200a82c6: stopping tablet replica
I20260929 16:34:59.015079 11502 tablet_replica.cc:330] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f: stopping tablet replica
I20260929 16:34:59.015400 11502 raft_consensus.cc:2229] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:34:59.015691 11502 raft_consensus.cc:2258] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:59.016077 11500 tablet_replica.cc:330] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706: stopping tablet replica
I20260929 16:34:59.016335 11500 raft_consensus.cc:2229] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:34:59.016649 11500 raft_consensus.cc:2258] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:34:59.018167 11502 ts_tablet_manager.cc:1905] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:59.018177 11500 ts_tablet_manager.cc:1905] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:34:59.021085 11500 ts_tablet_manager.cc:1918] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:59.021296 11500 log.cc:1196] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-1/wal/wals/3362652a3c274eecb35285d3d453560a
I20260929 16:34:59.021669 11500 ts_tablet_manager.cc:1939] T 3362652a3c274eecb35285d3d453560a P f8297aa01f3e47339fe1a0227c002706: Deleting consensus metadata
I20260929 16:34:59.021795 11502 ts_tablet_manager.cc:1918] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:34:59.021896 11502 log.cc:1196] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f: Deleting WAL directory at /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699450427356-23728-0/minicluster-data/ts-2/wal/wals/3362652a3c274eecb35285d3d453560a
I20260929 16:34:59.022225 11502 ts_tablet_manager.cc:1939] T 3362652a3c274eecb35285d3d453560a P 814f2149ac9a46ec88b37df6dee6741f: Deleting consensus metadata
W20260929 16:34:59.022315 10998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.44.1:40511 (error 108)
I20260929 16:34:59.023068 10996 catalog_manager.cc:4841] TS f8297aa01f3e47339fe1a0227c002706 (127.23.44.2:35695): tablet 3362652a3c274eecb35285d3d453560a (table test-table [id=2e6f2eea05be474581722ebcbdc262c1]) successfully deleted
I20260929 16:34:59.023392 10995 catalog_manager.cc:4841] TS 814f2149ac9a46ec88b37df6dee6741f (127.23.44.3:37597): tablet 3362652a3c274eecb35285d3d453560a (table test-table [id=2e6f2eea05be474581722ebcbdc262c1]) successfully deleted
I20260929 16:34:59.023653 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 11202
I20260929 16:34:59.023943 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:59.023681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.1:40511 (local address 127.0.0.1:53572)
0929 16:34:59.023807 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:59.023871 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.1:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:34:59.024389 10998 catalog_manager.cc:4584] TS cdf6fade847a448f80a2fd5f200a82c6 (127.23.44.1:40511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3362652a3c274eecb35285d3d453560a: Network error: Client connection negotiation failed: client connection to 127.23.44.1:40511: connect: Connection refused (error 111)
I20260929 16:34:59.029182 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 11333
I20260929 16:34:59.034845 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 10980
2026-09-29T16:34:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1208 ms)
[----------] 6 tests from EncryptionPolicyTest (10436 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:59 dist-test-slave-mlmz krb5kdc[11506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:59 dist-test-slave-mlmz krb5kdc[11506](info): set up 2 sockets
Sep 29 16:34:59 dist-test-slave-mlmz krb5kdc[11506](info): commencing operation
krb5kdc: starting...
W20260929 16:35:00.069272 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:00 dist-test-slave-mlmz krb5kdc[11506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:35:00.188001 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:00.293496 11519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:00.293759 11519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:00.293866 11519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:00.298413 11519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:00.298494 11519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:00.298533 11519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:00.298568 11519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:00.298602 11519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:00.303685 11519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:00.304662 11519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:00.305634 11519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:00.313645 11527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:00.313606 11525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:00.313606 11524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:00.314116 11519 server_base.cc:1034] running on GCE node
I20260929 16:35:00.314668 11519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:00.315685 11519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:00.316916 11519 hybrid_clock.cc:648] HybridClock initialized: now 1790699700316905 us; error 28 us; skew 500 ppm
Sep 29 16:35:00 dist-test-slave-mlmz krb5kdc[11506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:00.325757 11519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:35:00.326555 11519 webserver.cc:460] Webserver started at http://127.0.0.1:37651/ using document root <none> and password file <none>
I20260929 16:35:00.327059 11519 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:00.327116 11519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:00.327277 11519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:00.329178 11519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "afc062102b1d43c9a0d26b7e550c9064"
format_stamp: "Formatted at 2026-09-29 16:35:00 on dist-test-slave-mlmz"
server_key: "4ee68762e3358ed1bbb21adf0752c8a4"
server_key_iv: "3e8eff415ce91c51f52daf29a149a6f3"
server_key_version: "encryptionkey@0"
I20260929 16:35:00.329734 11519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "afc062102b1d43c9a0d26b7e550c9064"
format_stamp: "Formatted at 2026-09-29 16:35:00 on dist-test-slave-mlmz"
server_key: "4ee68762e3358ed1bbb21adf0752c8a4"
server_key_iv: "3e8eff415ce91c51f52daf29a149a6f3"
server_key_version: "encryptionkey@0"
I20260929 16:35:00.333096 11519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:35:00.335361 11534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:00.336519 11519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:00.336613 11519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "afc062102b1d43c9a0d26b7e550c9064"
format_stamp: "Formatted at 2026-09-29 16:35:00 on dist-test-slave-mlmz"
server_key: "4ee68762e3358ed1bbb21adf0752c8a4"
server_key_iv: "3e8eff415ce91c51f52daf29a149a6f3"
server_key_version: "encryptionkey@0"
I20260929 16:35:00.336737 11519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:00.368788 11519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:00.371891 11519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:00.372087 11519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:00.378824 11519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34661
I20260929 16:35:00.378851 11586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34661 every 8 connection(s)
I20260929 16:35:00.379717 11519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:00.382300 11587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:00.383424 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 11519
I20260929 16:35:00.383601 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:00.383878 23728 external_mini_cluster.cc:1428] Setting key 64ccad48c91fa4fb919830f52d78e28e
I20260929 16:35:00.387892 11587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064: Bootstrap starting.
I20260929 16:35:00.389928 11587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:00.390587 11587 log.cc:824] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:35:00 dist-test-slave-mlmz krb5kdc[11506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:35:00.392496 11587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064: No bootstrap required, opened a new log
I20260929 16:35:00.395265 11587 raft_consensus.cc:348] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc062102b1d43c9a0d26b7e550c9064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34661 } }
I20260929 16:35:00.395457 11587 raft_consensus.cc:374] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:00.395524 11587 raft_consensus.cc:729] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc062102b1d43c9a0d26b7e550c9064, State: Initialized, Role: FOLLOWER
I20260929 16:35:00.395943 11587 consensus_queue.cc:260] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc062102b1d43c9a0d26b7e550c9064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34661 } }
I20260929 16:35:00.396080 11587 raft_consensus.cc:388] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:00.396145 11587 raft_consensus.cc:482] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:00.396248 11587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:00.397294 11587 raft_consensus.cc:504] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc062102b1d43c9a0d26b7e550c9064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34661 } }
I20260929 16:35:00.397631 11587 leader_election.cc:302] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afc062102b1d43c9a0d26b7e550c9064; no voters:
I20260929 16:35:00.397920 11587 leader_election.cc:288] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:00.398056 11592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:00.398307 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:00.385251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36201 (local address 127.0.0.1:34661)
0929 16:35:00.385686 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:00.385700 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:00.385732 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:35:00.386351 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:00.386366 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:00.386669 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:00.387023 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:00.387816 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:00.388788 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:00.389699 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:00.390204 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:00.390371 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:00.393071 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:00.393076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:00.393088 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:00.393111 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:00.395181 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:00.395763 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:00.395768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:00.395772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:00.395870 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:00.396209 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:00.396212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:00.396214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:00.396703 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:00.396929 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:00.397140 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 16:35:00.397385 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260929 16:35:00.398291 11592 raft_consensus.cc:686] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [term 1 LEADER]: Becoming Leader. State: Replica: afc062102b1d43c9a0d26b7e550c9064, State: Running, Role: LEADER
I20260929 16:35:00.399183 11587 sys_catalog.cc:564] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:00.399412 11592 consensus_queue.cc:237] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc062102b1d43c9a0d26b7e550c9064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34661 } }
I20260929 16:35:00.400926 11593 sys_catalog.cc:455] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afc062102b1d43c9a0d26b7e550c9064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc062102b1d43c9a0d26b7e550c9064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34661 } } }
I20260929 16:35:00.401051 11593 sys_catalog.cc:458] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:00.401409 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:00.401381 11594 sys_catalog.cc:455] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afc062102b1d43c9a0d26b7e550c9064. Latest consensus state: current_term: 1 leader_uuid: "afc062102b1d43c9a0d26b7e550c9064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc062102b1d43c9a0d26b7e550c9064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34661 } } }
I20260929 16:35:00.401579 11594 sys_catalog.cc:458] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:00.401759 11607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:00.404631 11607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:00.413548 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:00.403399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40543 (local address 127.0.0.1:34661)
0929 16:35:00.403518 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:00.403522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:00.405846 (+ 2324us) server_negotiation.cc:407] Connection header received
0929 16:35:00.405994 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:00.405997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:00.406042 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:00.406157 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:00.406896 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:00.407675 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:00.408741 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:00.408972 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:00.409079 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:00.410413 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:00.410417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:00.410419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:00.410442 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:00.411834 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:00.412292 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:00.412296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:00.412297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:00.412375 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:00.412700 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:00.412703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:00.412704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:00.412864 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:00.413009 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:00.413295 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:35:00.413382 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:35:00.414759 11607 catalog_manager.cc:1347] Generated new cluster ID: 4b92899292d246c38df76e9180140051
I20260929 16:35:00.414834 11607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:00.433768 11607 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:00.434862 11607 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:00.443406 11607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afc062102b1d43c9a0d26b7e550c9064: Generated new TSK 0
I20260929 16:35:00.443991 11607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:00.454556 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 11519
W20260929 16:35:00.459784 11504 connection.cc:535] client connection to 127.0.0.1:34661 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34661): Connection reset by peer (error 104)
2026-09-29T16:35:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:00 dist-test-slave-mlmz krb5kdc[11618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:00 dist-test-slave-mlmz krb5kdc[11618](info): set up 2 sockets
Sep 29 16:35:00 dist-test-slave-mlmz krb5kdc[11618](info): commencing operation
krb5kdc: starting...
W20260929 16:35:01.485539 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:01 dist-test-slave-mlmz krb5kdc[11618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:35:01.608951 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:01.715574 11631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:01.715875 11631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:01.715957 11631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:01.719901 11631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:01.719985 11631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:01.720022 11631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:01.720081 11631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:01.720120 11631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:01.725464 11631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45501
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:01.726467 11631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:01.727452 11631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:01.735483 11637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:01.735483 11639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:01.735533 11636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:01.735821 11631 server_base.cc:1034] running on GCE node
I20260929 16:35:01.736275 11631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:01.737232 11631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:01.738456 11631 hybrid_clock.cc:648] HybridClock initialized: now 1790699701738425 us; error 43 us; skew 500 ppm
Sep 29 16:35:01 dist-test-slave-mlmz krb5kdc[11618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:01.741158 11631 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:35:01.742640 11631 webserver.cc:460] Webserver started at http://127.0.0.1:44005/ using document root <none> and password file <none>
I20260929 16:35:01.743151 11631 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:01.743207 11631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:01.743391 11631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:01.745378 11631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "f670d1afb7cc455f8f6abb10a14c1c4f"
format_stamp: "Formatted at 2026-09-29 16:35:01 on dist-test-slave-mlmz"
server_key: "b86b67ab6001a082bb83e45a90ee3f78"
server_key_iv: "d3147fc3a835b89796eba062faf8cb3d"
server_key_version: "encryptionkey@0"
I20260929 16:35:01.745884 11631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "f670d1afb7cc455f8f6abb10a14c1c4f"
format_stamp: "Formatted at 2026-09-29 16:35:01 on dist-test-slave-mlmz"
server_key: "b86b67ab6001a082bb83e45a90ee3f78"
server_key_iv: "d3147fc3a835b89796eba062faf8cb3d"
server_key_version: "encryptionkey@0"
I20260929 16:35:01.749300 11631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:35:01.751549 11646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:01.752765 11631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:01.752862 11631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "f670d1afb7cc455f8f6abb10a14c1c4f"
format_stamp: "Formatted at 2026-09-29 16:35:01 on dist-test-slave-mlmz"
server_key: "b86b67ab6001a082bb83e45a90ee3f78"
server_key_iv: "d3147fc3a835b89796eba062faf8cb3d"
server_key_version: "encryptionkey@0"
I20260929 16:35:01.752944 11631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:01.767475 11631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:01.770212 11631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:01.770366 11631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:01.777033 11631 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45501
I20260929 16:35:01.777099 11698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45501 every 8 connection(s)
I20260929 16:35:01.777992 11631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:01.780663 11699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:01.784688 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 11631
I20260929 16:35:01.784842 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:01.785113 23728 external_mini_cluster.cc:1428] Setting key 92414d814a2b8aa891a9ce70bac41552
I20260929 16:35:01.786281 11699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f: Bootstrap starting.
I20260929 16:35:01.788877 11699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:01.789754 11699 log.cc:824] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:01.791714 11699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f: No bootstrap required, opened a new log
I20260929 16:35:01.795171 11702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:01.786509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35575 (local address 127.0.0.1:45501)
0929 16:35:01.786913 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:01.786931 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:01.786970 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:35:01.787798 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:01.787819 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:01.788218 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:01.788569 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:01.788981 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:01.790361 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:01.791297 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:01.791963 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:01.792130 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:01.793016 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:01.793022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:01.793038 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:01.793576 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:01.793645 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:01.793675 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:01.793815 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:01.794019 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 16:35:01.794151 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260929 16:35:01.796178 11699 raft_consensus.cc:348] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f670d1afb7cc455f8f6abb10a14c1c4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } }
I20260929 16:35:01.796383 11699 raft_consensus.cc:374] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:01.796440 11699 raft_consensus.cc:729] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f670d1afb7cc455f8f6abb10a14c1c4f, State: Initialized, Role: FOLLOWER
I20260929 16:35:01.796836 11699 consensus_queue.cc:260] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f670d1afb7cc455f8f6abb10a14c1c4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } }
I20260929 16:35:01.796984 11699 raft_consensus.cc:388] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:01.797076 11699 raft_consensus.cc:482] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:01.797180 11699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:01.798148 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:01.798249 11699 raft_consensus.cc:504] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f670d1afb7cc455f8f6abb10a14c1c4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } }
I20260929 16:35:01.798585 11699 leader_election.cc:302] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f670d1afb7cc455f8f6abb10a14c1c4f; no voters:
I20260929 16:35:01.798933 11699 leader_election.cc:288] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:01.799033 11704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:01.799254 11704 raft_consensus.cc:686] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [term 1 LEADER]: Becoming Leader. State: Replica: f670d1afb7cc455f8f6abb10a14c1c4f, State: Running, Role: LEADER
I20260929 16:35:01.799589 11704 consensus_queue.cc:237] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f670d1afb7cc455f8f6abb10a14c1c4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } }
I20260929 16:35:01.800146 11699 sys_catalog.cc:564] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:01.800791 11704 sys_catalog.cc:455] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f670d1afb7cc455f8f6abb10a14c1c4f. Latest consensus state: current_term: 1 leader_uuid: "f670d1afb7cc455f8f6abb10a14c1c4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f670d1afb7cc455f8f6abb10a14c1c4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } } }
I20260929 16:35:01.800923 11704 sys_catalog.cc:458] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:01.801335 11705 sys_catalog.cc:455] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f670d1afb7cc455f8f6abb10a14c1c4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f670d1afb7cc455f8f6abb10a14c1c4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } } }
I20260929 16:35:01.801437 11705 sys_catalog.cc:458] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:01.804818 11702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:01.800387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56673 (local address 127.0.0.1:45501)
0929 16:35:01.800517 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:01.800521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:01.800748 (+ 227us) server_negotiation.cc:407] Connection header received
0929 16:35:01.800929 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:01.800935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:01.800983 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:01.801036 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:01.801535 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:01.802064 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:01.803050 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:01.803362 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:01.803482 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:01.804334 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:01.804337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:01.804340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:01.804352 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:01.804383 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:01.804393 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:01.804493 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:01.804629 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 16:35:01.804664 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 16:35:01.805763 11724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:35:01.805845 11724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:35:01.805939 11715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:01.806638 11715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:01.813141 11715 catalog_manager.cc:1347] Generated new cluster ID: e155df70fe6f41f58c3b25e59ac020ee
I20260929 16:35:01.813203 11715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:01.819985 11715 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:01.820832 11715 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:01.826526 11715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f670d1afb7cc455f8f6abb10a14c1c4f: Generated new TSK 0
I20260929 16:35:01.827152 11715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:01.831339 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 11631
W20260929 16:35:01.836570 11616 connection.cc:535] client connection to 127.0.0.1:45501 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45501): Connection reset by peer (error 104)
2026-09-29T16:35:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:01 dist-test-slave-mlmz krb5kdc[11730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:01 dist-test-slave-mlmz krb5kdc[11730](info): set up 2 sockets
Sep 29 16:35:01 dist-test-slave-mlmz krb5kdc[11730](info): commencing operation
krb5kdc: starting...
W20260929 16:35:02.877351 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:02 dist-test-slave-mlmz krb5kdc[11730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:35:02.995564 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:03.099601 11743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:03.099858 11743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:03.099946 11743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:03.103860 11743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:03.103937 11743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:03.103982 11743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:03.104040 11743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:03.104089 11743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:03.109109 11743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:03.110069 11743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:03.110992 11743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:03.118880 11749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:03.118886 11751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:03.118961 11748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:03.119086 11743 server_base.cc:1034] running on GCE node
I20260929 16:35:03.119665 11743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:03.120687 11743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:03.121878 11743 hybrid_clock.cc:648] HybridClock initialized: now 1790699703121863 us; error 27 us; skew 500 ppm
Sep 29 16:35:03 dist-test-slave-mlmz krb5kdc[11730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:03.124584 11743 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:35:03.125577 11743 webserver.cc:460] Webserver started at http://127.0.0.1:44737/ using document root <none> and password file <none>
I20260929 16:35:03.126104 11743 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:03.126190 11743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:03.126416 11743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:03.128357 11743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "9f2ebfd73eca4874b65865bd6f19c339"
format_stamp: "Formatted at 2026-09-29 16:35:03 on dist-test-slave-mlmz"
server_key: "8867e113fba4bad86bc2dfa7677dd3a7"
server_key_iv: "ff9848414870bcca5ccde5f25517ae57"
server_key_version: "encryptionkey@0"
I20260929 16:35:03.128906 11743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "9f2ebfd73eca4874b65865bd6f19c339"
format_stamp: "Formatted at 2026-09-29 16:35:03 on dist-test-slave-mlmz"
server_key: "8867e113fba4bad86bc2dfa7677dd3a7"
server_key_iv: "ff9848414870bcca5ccde5f25517ae57"
server_key_version: "encryptionkey@0"
I20260929 16:35:03.132395 11743 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:35:03.134784 11758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:03.135977 11743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:35:03.136116 11743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "9f2ebfd73eca4874b65865bd6f19c339"
format_stamp: "Formatted at 2026-09-29 16:35:03 on dist-test-slave-mlmz"
server_key: "8867e113fba4bad86bc2dfa7677dd3a7"
server_key_iv: "ff9848414870bcca5ccde5f25517ae57"
server_key_version: "encryptionkey@0"
I20260929 16:35:03.136224 11743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:03.140616 11743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:03.143800 11743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:03.143990 11743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:03.151034 11743 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45883
I20260929 16:35:03.151058 11810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45883 every 8 connection(s)
I20260929 16:35:03.151880 11743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:03.154407 11811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:03.159524 11811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339: Bootstrap starting.
I20260929 16:35:03.160768 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 11743
I20260929 16:35:03.160913 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:03.161213 23728 external_mini_cluster.cc:1428] Setting key a24dcb39d18e90f241e8f58d4d57f98d
I20260929 16:35:03.162213 11811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:03.162984 11811 log.cc:824] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:03.164896 11811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339: No bootstrap required, opened a new log
I20260929 16:35:03.166146 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:03.162291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48395 (local address 127.0.0.1:45883)
0929 16:35:03.162718 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:03.162732 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:03.162774 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:35:03.163543 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:03.163566 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:03.163907 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:03.164240 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:03.165019 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:03.165024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:03.165035 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:03.165045 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:03.165095 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:03.165113 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:03.165227 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:03.165371 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 16:35:03.165525 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
I20260929 16:35:03.168852 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:03.168972 11811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2ebfd73eca4874b65865bd6f19c339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45883 } }
I20260929 16:35:03.169214 11811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:03.169284 11811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2ebfd73eca4874b65865bd6f19c339, State: Initialized, Role: FOLLOWER
I20260929 16:35:03.169713 11811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2ebfd73eca4874b65865bd6f19c339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45883 } }
I20260929 16:35:03.169857 11811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:03.169922 11811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:03.170015 11811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:03.170938 11811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2ebfd73eca4874b65865bd6f19c339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45883 } }
I20260929 16:35:03.171353 11811 leader_election.cc:302] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f2ebfd73eca4874b65865bd6f19c339; no voters:
I20260929 16:35:03.171712 11811 leader_election.cc:288] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:03.171818 11821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:03.172025 11821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [term 1 LEADER]: Becoming Leader. State: Replica: 9f2ebfd73eca4874b65865bd6f19c339, State: Running, Role: LEADER
I20260929 16:35:03.172350 11821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2ebfd73eca4874b65865bd6f19c339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45883 } }
I20260929 16:35:03.172632 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:03.170957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33651 (local address 127.0.0.1:45883)
0929 16:35:03.171079 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:03.171083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:03.171247 (+ 164us) server_negotiation.cc:407] Connection header received
0929 16:35:03.171352 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:03.171354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:03.171379 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:03.171441 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:03.172205 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:03.172208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:03.172211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:03.172222 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:03.172248 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:03.172260 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:03.172312 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:03.172479 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 16:35:03.172520 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:35:03.172978 11811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:03.173754 11821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f2ebfd73eca4874b65865bd6f19c339. Latest consensus state: current_term: 1 leader_uuid: "9f2ebfd73eca4874b65865bd6f19c339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2ebfd73eca4874b65865bd6f19c339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45883 } } }
I20260929 16:35:03.173933 11821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:03.174166 11822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f2ebfd73eca4874b65865bd6f19c339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2ebfd73eca4874b65865bd6f19c339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45883 } } }
I20260929 16:35:03.174295 11822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:03.174373 11830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:03.177340 11830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:03.185073 11830 catalog_manager.cc:1347] Generated new cluster ID: fd2d0261a12940cb958d52a94a857535
I20260929 16:35:03.185140 11830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:03.203313 11830 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:03.203595 11830 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:03.211553 11830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f2ebfd73eca4874b65865bd6f19c339: Generated new TSK 0
I20260929 16:35:03.212349 11830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:03.222062 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 11743
2026-09-29T16:35:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:03 dist-test-slave-mlmz krb5kdc[11842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:03 dist-test-slave-mlmz krb5kdc[11842](info): set up 2 sockets
Sep 29 16:35:03 dist-test-slave-mlmz krb5kdc[11842](info): commencing operation
krb5kdc: starting...
W20260929 16:35:04.293462 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:04 dist-test-slave-mlmz krb5kdc[11842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:35:04.412256 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:04.517095 11855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:04.517412 11855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:04.517491 11855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:04.521404 11855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:04.521488 11855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:04.521525 11855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:04.521581 11855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:04.521620 11855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:04.526685 11855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:04.527652 11855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:04.528699 11855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:04.536959 11861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:04.536979 11863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:04.536979 11860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:04.537171 11855 server_base.cc:1034] running on GCE node
I20260929 16:35:04.537703 11855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:04.538702 11855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:04.539892 11855 hybrid_clock.cc:648] HybridClock initialized: now 1790699704539878 us; error 34 us; skew 500 ppm
Sep 29 16:35:04 dist-test-slave-mlmz krb5kdc[11842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:04.542552 11855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:35:04.543355 11855 webserver.cc:460] Webserver started at http://127.0.0.1:44719/ using document root <none> and password file <none>
I20260929 16:35:04.543886 11855 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:04.543979 11855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:04.544210 11855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:04.546195 11855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "448460c8c7ee43a8a913308619342c64"
format_stamp: "Formatted at 2026-09-29 16:35:04 on dist-test-slave-mlmz"
server_key: "6f04a7c16a8e108a91cef5775fa5b1cf"
server_key_iv: "6d4503d0ad261a0aabd934fcf43188de"
server_key_version: "encryptionkey@0"
I20260929 16:35:04.546868 11855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "448460c8c7ee43a8a913308619342c64"
format_stamp: "Formatted at 2026-09-29 16:35:04 on dist-test-slave-mlmz"
server_key: "6f04a7c16a8e108a91cef5775fa5b1cf"
server_key_iv: "6d4503d0ad261a0aabd934fcf43188de"
server_key_version: "encryptionkey@0"
I20260929 16:35:04.551815 11855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:35:04.554282 11870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:04.555765 11855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:04.555888 11855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "448460c8c7ee43a8a913308619342c64"
format_stamp: "Formatted at 2026-09-29 16:35:04 on dist-test-slave-mlmz"
server_key: "6f04a7c16a8e108a91cef5775fa5b1cf"
server_key_iv: "6d4503d0ad261a0aabd934fcf43188de"
server_key_version: "encryptionkey@0"
I20260929 16:35:04.555994 11855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:04.582449 11855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:04.585309 11855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:04.585505 11855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:04.592363 11855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42583
I20260929 16:35:04.592389 11922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42583 every 8 connection(s)
I20260929 16:35:04.593287 11855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:04.596441 11923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:04.597860 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 11855
I20260929 16:35:04.598011 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:04.598316 23728 external_mini_cluster.cc:1428] Setting key 452e8deb40a43aa0bbe4df5d758f9be5
I20260929 16:35:04.602519 11923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64: Bootstrap starting.
I20260929 16:35:04.604647 11923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:04.605521 11923 log.cc:824] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:35:04 dist-test-slave-mlmz krb5kdc[11842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:35:04.607501 11923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64: No bootstrap required, opened a new log
I20260929 16:35:04.610826 11923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448460c8c7ee43a8a913308619342c64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } }
I20260929 16:35:04.611157 11923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:04.611248 11923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448460c8c7ee43a8a913308619342c64, State: Initialized, Role: FOLLOWER
I20260929 16:35:04.611671 11923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448460c8c7ee43a8a913308619342c64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } }
I20260929 16:35:04.611817 11923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:04.611902 11923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:04.611969 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:04.599711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55249 (local address 127.0.0.1:42583)
0929 16:35:04.600118 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:04.600132 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:04.600165 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:35:04.600776 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:04.600790 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:04.601104 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:04.601463 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:04.602261 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:04.603248 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:04.604097 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:04.604550 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:04.604725 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:04.607334 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:04.607339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:04.607351 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:04.607377 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:04.609041 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:04.609537 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:04.609542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:04.609547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:04.609650 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:04.609937 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:04.609940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:04.609942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:04.610316 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:04.610527 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:04.610757 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:35:04.611036 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260929 16:35:04.611997 11923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:04.613078 11923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448460c8c7ee43a8a913308619342c64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } }
I20260929 16:35:04.613433 11923 leader_election.cc:302] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 448460c8c7ee43a8a913308619342c64; no voters:
I20260929 16:35:04.613772 11923 leader_election.cc:288] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:04.613876 11928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:04.614099 11928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [term 1 LEADER]: Becoming Leader. State: Replica: 448460c8c7ee43a8a913308619342c64, State: Running, Role: LEADER
I20260929 16:35:04.614421 11928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448460c8c7ee43a8a913308619342c64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } }
I20260929 16:35:04.614871 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:04.614982 11923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:04.616019 11930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "448460c8c7ee43a8a913308619342c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448460c8c7ee43a8a913308619342c64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } } }
I20260929 16:35:04.616144 11930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:04.616834 11929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 448460c8c7ee43a8a913308619342c64. Latest consensus state: current_term: 1 leader_uuid: "448460c8c7ee43a8a913308619342c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448460c8c7ee43a8a913308619342c64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } } }
I20260929 16:35:04.616933 11929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:04.617617 11940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:04.620783 11940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:04.626436 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:04.617649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50817 (local address 127.0.0.1:42583)
0929 16:35:04.617763 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:04.617767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:04.618019 (+ 252us) server_negotiation.cc:407] Connection header received
0929 16:35:04.618131 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:04.618133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:04.618167 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:04.618234 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:04.619032 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:04.619771 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:04.620860 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:04.621166 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:04.621293 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:04.623072 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:04.623075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:04.623078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:04.623102 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:04.624575 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:04.625059 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:04.625062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:04.625064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:04.625146 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:04.625502 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:04.625505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:04.625507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:04.625750 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:04.625876 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:04.626163 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 16:35:04.626278 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:35:04.627220 11940 catalog_manager.cc:1347] Generated new cluster ID: 2f62f4faa902444a93fd950d8607adc1
I20260929 16:35:04.627395 11940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:04.632623 11940 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:04.633744 11940 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:04.645500 11940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 448460c8c7ee43a8a913308619342c64: Generated new TSK 0
I20260929 16:35:04.646112 11940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:04.652685 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 11855
2026-09-29T16:35:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:04 dist-test-slave-mlmz krb5kdc[11954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:04 dist-test-slave-mlmz krb5kdc[11954](info): set up 2 sockets
Sep 29 16:35:04 dist-test-slave-mlmz krb5kdc[11954](info): commencing operation
krb5kdc: starting...
W20260929 16:35:05.745497 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:05 dist-test-slave-mlmz krb5kdc[11954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:35:05.860755 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:05.965284 11967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:05.965523 11967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:05.965564 11967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:05.969512 11967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:05.969563 11967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:05.969583 11967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:05.969599 11967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:05.969614 11967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:05.974597 11967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:05.975471 11967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:05.976361 11967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:05.984344 11972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:05.984344 11973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:05.984364 11975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:05.984913 11967 server_base.cc:1034] running on GCE node
I20260929 16:35:05.985548 11967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:05.986559 11967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:05.987715 11967 hybrid_clock.cc:648] HybridClock initialized: now 1790699705987697 us; error 29 us; skew 500 ppm
Sep 29 16:35:05 dist-test-slave-mlmz krb5kdc[11954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:05.994019 11967 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:35:05.994915 11967 webserver.cc:460] Webserver started at http://127.0.0.1:32937/ using document root <none> and password file <none>
I20260929 16:35:05.995440 11967 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:05.995500 11967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:05.995708 11967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:05.997678 11967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "9f21f97d6f4642fb9871b6cbd83bee77"
format_stamp: "Formatted at 2026-09-29 16:35:05 on dist-test-slave-mlmz"
server_key: "2f253bb0b5d5c1cf3726c56800f2ee9f"
server_key_iv: "6d4e17ddd9d5bc3ea131c5bb5b2d2707"
server_key_version: "encryptionkey@0"
I20260929 16:35:05.998208 11967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "9f21f97d6f4642fb9871b6cbd83bee77"
format_stamp: "Formatted at 2026-09-29 16:35:05 on dist-test-slave-mlmz"
server_key: "2f253bb0b5d5c1cf3726c56800f2ee9f"
server_key_iv: "6d4e17ddd9d5bc3ea131c5bb5b2d2707"
server_key_version: "encryptionkey@0"
I20260929 16:35:06.001659 11967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:35:06.003859 11982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:06.005019 11967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:06.005157 11967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "9f21f97d6f4642fb9871b6cbd83bee77"
format_stamp: "Formatted at 2026-09-29 16:35:05 on dist-test-slave-mlmz"
server_key: "2f253bb0b5d5c1cf3726c56800f2ee9f"
server_key_iv: "6d4e17ddd9d5bc3ea131c5bb5b2d2707"
server_key_version: "encryptionkey@0"
I20260929 16:35:06.005263 11967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:06.029945 11967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:06.032913 11967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:06.033113 11967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:06.040256 11967 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44065
I20260929 16:35:06.040280 12034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44065 every 8 connection(s)
I20260929 16:35:06.041123 11967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:06.043632 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:06.046252 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 11967
I20260929 16:35:06.046389 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:06.046662 23728 external_mini_cluster.cc:1428] Setting key 050f119a9fffebe51d0cef422ad8c4b5
I20260929 16:35:06.049594 12035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77: Bootstrap starting.
I20260929 16:35:06.052139 12035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:06.052924 12035 log.cc:824] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:06.055094 12035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77: No bootstrap required, opened a new log
I20260929 16:35:06.056209 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.047952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50547 (local address 127.0.0.1:44065)
0929 16:35:06.048361 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.048381 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.048425 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:35:06.049180 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.049201 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.049628 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.049973 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:06.050367 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.051824 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.052721 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.053416 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.053536 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.054376 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:06.054382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:06.054396 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:06.054855 (+ 459us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:06.054919 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:06.054948 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:06.055045 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.055177 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.055377 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":95,"threads_started":1}
I20260929 16:35:06.058969 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:06.059211 12035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f21f97d6f4642fb9871b6cbd83bee77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } }
I20260929 16:35:06.059409 12035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:06.059484 12035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21f97d6f4642fb9871b6cbd83bee77, State: Initialized, Role: FOLLOWER
I20260929 16:35:06.059880 12035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f21f97d6f4642fb9871b6cbd83bee77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } }
I20260929 16:35:06.059993 12035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:06.060078 12035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:06.060192 12035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:06.061316 12035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f21f97d6f4642fb9871b6cbd83bee77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } }
I20260929 16:35:06.061671 12035 leader_election.cc:302] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f21f97d6f4642fb9871b6cbd83bee77; no voters:
I20260929 16:35:06.062000 12035 leader_election.cc:288] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:06.062176 12045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:06.062436 12045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [term 1 LEADER]: Becoming Leader. State: Replica: 9f21f97d6f4642fb9871b6cbd83bee77, State: Running, Role: LEADER
I20260929 16:35:06.062749 12045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f21f97d6f4642fb9871b6cbd83bee77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } }
I20260929 16:35:06.063181 12035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:06.063864 12045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f21f97d6f4642fb9871b6cbd83bee77. Latest consensus state: current_term: 1 leader_uuid: "9f21f97d6f4642fb9871b6cbd83bee77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f21f97d6f4642fb9871b6cbd83bee77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } } }
I20260929 16:35:06.064301 12046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f21f97d6f4642fb9871b6cbd83bee77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f21f97d6f4642fb9871b6cbd83bee77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44065 } } }
I20260929 16:35:06.064575 12045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:06.064620 12046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:06.065016 12055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:06.065963 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:06.061090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38319 (local address 127.0.0.1:44065)
0929 16:35:06.061215 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:06.061219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:06.061325 (+ 106us) server_negotiation.cc:407] Connection header received
0929 16:35:06.061473 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:06.061477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:06.061521 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:06.061605 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:06.061858 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.062633 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.063737 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:06.064139 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:06.064252 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:06.065236 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:06.065239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:06.065242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:06.065275 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:06.065306 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:06.065321 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:06.065386 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:06.065796 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 16:35:06.065844 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:35:06.067885 12055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:06.073912 12055 catalog_manager.cc:1347] Generated new cluster ID: c9095ab400154ab78f9c11062833fcfb
I20260929 16:35:06.073972 12055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:06.100589 12055 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:06.101744 12055 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:06.109530 12055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f21f97d6f4642fb9871b6cbd83bee77: Generated new TSK 0
I20260929 16:35:06.110239 12055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:06.120981 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 11967
2026-09-29T16:35:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:06 dist-test-slave-mlmz krb5kdc[12066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:06 dist-test-slave-mlmz krb5kdc[12066](info): set up 2 sockets
Sep 29 16:35:06 dist-test-slave-mlmz krb5kdc[12066](info): commencing operation
krb5kdc: starting...
W20260929 16:35:07.153359 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:07 dist-test-slave-mlmz krb5kdc[12066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:35:07.276401 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:07.380620 12079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:07.380857 12079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:07.380896 12079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:07.385100 12079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:07.385170 12079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:07.385205 12079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:07.385246 12079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:07.385295 12079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:07.390514 12079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:07.391458 12079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:07.392403 12079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:07.401023 12084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:07.401000 12085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:07.401207 12079 server_base.cc:1034] running on GCE node
W20260929 16:35:07.401038 12087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:07.401914 12079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:07.402915 12079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:07.404121 12079 hybrid_clock.cc:648] HybridClock initialized: now 1790699707404081 us; error 41 us; skew 500 ppm
Sep 29 16:35:07 dist-test-slave-mlmz krb5kdc[12066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:07.406855 12079 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:35:07.407723 12079 webserver.cc:460] Webserver started at http://127.0.0.1:33075/ using document root <none> and password file <none>
I20260929 16:35:07.408272 12079 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:07.408366 12079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:07.408586 12079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:07.410542 12079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "9264015dcf864b098467d6823300578d"
format_stamp: "Formatted at 2026-09-29 16:35:07 on dist-test-slave-mlmz"
server_key: "d0154cf440bccbe5a2538f8f93861d74"
server_key_iv: "7665b051abbe550416a2d96f47c79ea8"
server_key_version: "encryptionkey@0"
I20260929 16:35:07.411088 12079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "9264015dcf864b098467d6823300578d"
format_stamp: "Formatted at 2026-09-29 16:35:07 on dist-test-slave-mlmz"
server_key: "d0154cf440bccbe5a2538f8f93861d74"
server_key_iv: "7665b051abbe550416a2d96f47c79ea8"
server_key_version: "encryptionkey@0"
I20260929 16:35:07.414568 12079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:07.416754 12094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:07.418009 12079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:35:07.418149 12079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "9264015dcf864b098467d6823300578d"
format_stamp: "Formatted at 2026-09-29 16:35:07 on dist-test-slave-mlmz"
server_key: "d0154cf440bccbe5a2538f8f93861d74"
server_key_iv: "7665b051abbe550416a2d96f47c79ea8"
server_key_version: "encryptionkey@0"
I20260929 16:35:07.418273 12079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:07.422667 12079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:07.425519 12079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:07.425729 12079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:07.432785 12079 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38915
I20260929 16:35:07.432798 12146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38915 every 8 connection(s)
I20260929 16:35:07.433697 12079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:07.436247 12147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:07.441542 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 12079
I20260929 16:35:07.441684 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:07.441807 12147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d: Bootstrap starting.
I20260929 16:35:07.441980 23728 external_mini_cluster.cc:1428] Setting key fa3f66de6a96e1cf8879a5a5b9ac375e
I20260929 16:35:07.444386 12147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:07.445204 12147 log.cc:824] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:07.447157 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:07.443252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42749 (local address 127.0.0.1:38915)
0929 16:35:07.443684 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:07.443701 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:07.443745 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:35:07.444408 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:07.444424 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:07.444767 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:07.445060 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:07.445899 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:07.445906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:07.445922 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:07.445935 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:07.445999 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:07.446043 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:07.446159 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:07.446280 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 16:35:07.446451 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":147,"threads_started":1}
I20260929 16:35:07.447476 12147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d: No bootstrap required, opened a new log
I20260929 16:35:07.450039 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:07.451395 12147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9264015dcf864b098467d6823300578d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38915 } }
I20260929 16:35:07.451678 12147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:07.451748 12147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9264015dcf864b098467d6823300578d, State: Initialized, Role: FOLLOWER
I20260929 16:35:07.452207 12147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9264015dcf864b098467d6823300578d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38915 } }
I20260929 16:35:07.452381 12147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:07.452453 12147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:07.452577 12147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:07.453656 12147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9264015dcf864b098467d6823300578d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38915 } }
I20260929 16:35:07.454020 12147 leader_election.cc:302] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9264015dcf864b098467d6823300578d; no voters:
I20260929 16:35:07.454187 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:07.452343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59483 (local address 127.0.0.1:38915)
0929 16:35:07.452457 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:07.452464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:07.452736 (+ 272us) server_negotiation.cc:407] Connection header received
0929 16:35:07.452859 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:07.452865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:07.452904 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:07.452993 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:07.453760 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:07.453763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:07.453765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:07.453776 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:07.453801 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:07.453813 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:07.453895 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:07.454039 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 16:35:07.454073 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:35:07.454370 12147 leader_election.cc:288] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:07.454876 12157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:07.455129 12157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [term 1 LEADER]: Becoming Leader. State: Replica: 9264015dcf864b098467d6823300578d, State: Running, Role: LEADER
I20260929 16:35:07.455421 12157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9264015dcf864b098467d6823300578d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38915 } }
I20260929 16:35:07.455554 12147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:07.456828 12159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9264015dcf864b098467d6823300578d. Latest consensus state: current_term: 1 leader_uuid: "9264015dcf864b098467d6823300578d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9264015dcf864b098467d6823300578d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38915 } } }
I20260929 16:35:07.456951 12159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:07.457327 12158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9264015dcf864b098467d6823300578d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9264015dcf864b098467d6823300578d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38915 } } }
I20260929 16:35:07.457423 12158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d [sys.catalog]: This master's current role is: LEADER
W20260929 16:35:07.459987 12172 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:35:07.460088 12172 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:35:07.460178 12168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:07.460865 12168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:07.467710 12168 catalog_manager.cc:1347] Generated new cluster ID: d7b42ca6735a4533a26f8a044ed950a1
I20260929 16:35:07.467797 12168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:07.486745 12168 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:07.487078 12168 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:07.499886 12168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9264015dcf864b098467d6823300578d: Generated new TSK 0
I20260929 16:35:07.500584 12168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:07.504266 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 12079
2026-09-29T16:35:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:07 dist-test-slave-mlmz krb5kdc[12178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:07 dist-test-slave-mlmz krb5kdc[12178](info): set up 2 sockets
Sep 29 16:35:07 dist-test-slave-mlmz krb5kdc[12178](info): commencing operation
krb5kdc: starting...
W20260929 16:35:08.545347 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:08 dist-test-slave-mlmz krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:35:08.664198 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:08.767588 12191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:08.767807 12191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:08.767846 12191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:08.771636 12191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:08.771682 12191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:08.771700 12191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:08.771716 12191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:08.771730 12191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:08.776561 12191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:08.777396 12191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:08.778237 12191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:08.786506 12197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:08.786546 12196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:08.786546 12199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:08.786763 12191 server_base.cc:1034] running on GCE node
I20260929 16:35:08.787297 12191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:08.788337 12191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:08.789517 12191 hybrid_clock.cc:648] HybridClock initialized: now 1790699708789503 us; error 27 us; skew 500 ppm
Sep 29 16:35:08 dist-test-slave-mlmz krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:08.792146 12191 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:35:08.792977 12191 webserver.cc:460] Webserver started at http://127.0.0.1:45633/ using document root <none> and password file <none>
I20260929 16:35:08.793553 12191 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:08.793650 12191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:08.793866 12191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:08.795727 12191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "2ce45d68ebfb46b68d66203bd3635e36"
format_stamp: "Formatted at 2026-09-29 16:35:08 on dist-test-slave-mlmz"
server_key: "65480ab3cdccddabbb84a51a00238fdc"
server_key_iv: "6a645195e3dacf034ecd662c8668f5b8"
server_key_version: "encryptionkey@0"
I20260929 16:35:08.796260 12191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "2ce45d68ebfb46b68d66203bd3635e36"
format_stamp: "Formatted at 2026-09-29 16:35:08 on dist-test-slave-mlmz"
server_key: "65480ab3cdccddabbb84a51a00238fdc"
server_key_iv: "6a645195e3dacf034ecd662c8668f5b8"
server_key_version: "encryptionkey@0"
I20260929 16:35:08.799674 12191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 16:35:08.801939 12206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:08.803011 12191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:08.803153 12191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "2ce45d68ebfb46b68d66203bd3635e36"
format_stamp: "Formatted at 2026-09-29 16:35:08 on dist-test-slave-mlmz"
server_key: "65480ab3cdccddabbb84a51a00238fdc"
server_key_iv: "6a645195e3dacf034ecd662c8668f5b8"
server_key_version: "encryptionkey@0"
I20260929 16:35:08.803254 12191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:08.822934 12191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:08.825977 12191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:08.826171 12191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:08.832903 12191 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34837
I20260929 16:35:08.832929 12258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34837 every 8 connection(s)
I20260929 16:35:08.833773 12191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:08.836267 12259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:08.839390 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 12191
I20260929 16:35:08.839536 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:08.839792 23728 external_mini_cluster.cc:1428] Setting key 4f622099e7e6f78191ae8f302a09a5f6
I20260929 16:35:08.841939 12259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36: Bootstrap starting.
I20260929 16:35:08.844286 12259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:08.845084 12259 log.cc:824] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:08.846817 12259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36: No bootstrap required, opened a new log
Sep 29 16:35:08 dist-test-slave-mlmz krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:35:08.850139 12259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce45d68ebfb46b68d66203bd3635e36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } }
I20260929 16:35:08.850358 12259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:08.850436 12259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce45d68ebfb46b68d66203bd3635e36, State: Initialized, Role: FOLLOWER
I20260929 16:35:08.850881 12259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce45d68ebfb46b68d66203bd3635e36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } }
I20260929 16:35:08.851025 12259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:08.851123 12259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:08.851225 12259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:08.852268 12259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce45d68ebfb46b68d66203bd3635e36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } }
I20260929 16:35:08.852609 12259 leader_election.cc:302] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ce45d68ebfb46b68d66203bd3635e36; no voters:
I20260929 16:35:08.852933 12259 leader_election.cc:288] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:08.853049 12264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:08.853255 12264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [term 1 LEADER]: Becoming Leader. State: Replica: 2ce45d68ebfb46b68d66203bd3635e36, State: Running, Role: LEADER
I20260929 16:35:08.853561 12264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce45d68ebfb46b68d66203bd3635e36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } }
I20260929 16:35:08.853771 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:08.841125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:56533 (local address 127.0.0.1:34837)
0929 16:35:08.841533 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:08.841552 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:08.841593 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:35:08.842237 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:08.842254 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:08.842565 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:08.842825 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:08.843684 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:08.844694 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:08.845794 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:08.846256 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:08.846424 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:08.848809 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:08.848814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:08.848826 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:08.848850 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:08.850644 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:08.851157 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:08.851164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:08.851170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:08.851310 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:08.851656 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:08.851659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:08.851661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:08.852148 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:08.852380 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:08.852649 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 16:35:08.852860 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":160,"threads_started":1}
I20260929 16:35:08.854115 12259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:08.855111 12265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ce45d68ebfb46b68d66203bd3635e36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce45d68ebfb46b68d66203bd3635e36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } } }
I20260929 16:35:08.855233 12265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:08.855531 12266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ce45d68ebfb46b68d66203bd3635e36. Latest consensus state: current_term: 1 leader_uuid: "2ce45d68ebfb46b68d66203bd3635e36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce45d68ebfb46b68d66203bd3635e36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } } }
I20260929 16:35:08.855623 12266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:08.855872 12273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:08.856865 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:08.859244 12273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:08.865496 12273 catalog_manager.cc:1347] Generated new cluster ID: 3dfaf2c5b00c4488aaabec4c974c9d28
I20260929 16:35:08.865589 12273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:08.867761 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:08.858991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:60781 (local address 127.0.0.1:34837)
0929 16:35:08.859093 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:08.859098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:08.859301 (+ 203us) server_negotiation.cc:407] Connection header received
0929 16:35:08.859477 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:08.859480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:08.859530 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:08.859623 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:08.860450 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:08.861185 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:08.862091 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:08.862345 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:08.862455 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:08.863961 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:08.863965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:08.863968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:08.863993 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:08.865694 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:08.866269 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:08.866272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:08.866274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:08.866355 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:08.866766 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:08.866769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:08.866771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:08.866972 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:08.867083 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:08.867498 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 16:35:08.867609 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 16:35:08.875388 12273 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:08.876284 12273 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:08.888442 12273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ce45d68ebfb46b68d66203bd3635e36: Generated new TSK 0
I20260929 16:35:08.889065 12273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:08.898706 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 12191
W20260929 16:35:08.904079 12176 connection.cc:535] client connection to 127.0.0.1:34837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34837): Connection reset by peer (error 104)
2026-09-29T16:35:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:08 dist-test-slave-mlmz krb5kdc[12290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:08 dist-test-slave-mlmz krb5kdc[12290](info): set up 2 sockets
Sep 29 16:35:08 dist-test-slave-mlmz krb5kdc[12290](info): commencing operation
krb5kdc: starting...
W20260929 16:35:09.985457 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:10 dist-test-slave-mlmz krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:35:10.102456 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:10.206270 12303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:10.206594 12303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:10.206681 12303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:10.210552 12303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:10.210633 12303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:10.210675 12303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:10.210736 12303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:10.210774 12303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:10.215708 12303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:10.216651 12303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:10.217593 12303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:10.225666 12308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:10.225725 12311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:10.225666 12309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:10.225818 12303 server_base.cc:1034] running on GCE node
I20260929 16:35:10.226441 12303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:10.227484 12303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:10.228673 12303 hybrid_clock.cc:648] HybridClock initialized: now 1790699710228668 us; error 32 us; skew 500 ppm
Sep 29 16:35:10 dist-test-slave-mlmz krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:10.231331 12303 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:35:10.232168 12303 webserver.cc:460] Webserver started at http://127.0.0.1:33797/ using document root <none> and password file <none>
I20260929 16:35:10.232678 12303 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:10.232777 12303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:10.232992 12303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:10.234920 12303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "7069cf0cc34f43328ec773c9f689b53d"
format_stamp: "Formatted at 2026-09-29 16:35:10 on dist-test-slave-mlmz"
server_key: "5fc15428bd5a580d8b649147d70a937d"
server_key_iv: "3783633bc33b048a52863af2260647f1"
server_key_version: "encryptionkey@0"
I20260929 16:35:10.235463 12303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "7069cf0cc34f43328ec773c9f689b53d"
format_stamp: "Formatted at 2026-09-29 16:35:10 on dist-test-slave-mlmz"
server_key: "5fc15428bd5a580d8b649147d70a937d"
server_key_iv: "3783633bc33b048a52863af2260647f1"
server_key_version: "encryptionkey@0"
I20260929 16:35:10.238834 12303 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:35:10.241001 12318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:10.242189 12303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:10.242331 12303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "7069cf0cc34f43328ec773c9f689b53d"
format_stamp: "Formatted at 2026-09-29 16:35:10 on dist-test-slave-mlmz"
server_key: "5fc15428bd5a580d8b649147d70a937d"
server_key_iv: "3783633bc33b048a52863af2260647f1"
server_key_version: "encryptionkey@0"
I20260929 16:35:10.242435 12303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:10.265419 12303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:10.268163 12303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:10.268362 12303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:10.275296 12303 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39981
I20260929 16:35:10.275331 12370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39981 every 8 connection(s)
I20260929 16:35:10.276196 12303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:10.277479 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 12303
I20260929 16:35:10.277619 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:10.277904 23728 external_mini_cluster.cc:1428] Setting key 75eb7e0297707227a14ebb6dfd20b957
I20260929 16:35:10.279300 12371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:10.285871 12371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d: Bootstrap starting.
I20260929 16:35:10.287642 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:10.279168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:59699 (local address 127.0.0.1:39981)
0929 16:35:10.279606 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:10.279625 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:10.279664 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:35:10.280477 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:10.280497 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:10.280903 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:10.281287 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:10.281673 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.283198 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.284051 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.284745 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.284910 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:10.285736 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:10.285743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:10.285757 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:10.286230 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:10.286293 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:10.286322 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:10.286450 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:10.286619 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 16:35:10.286740 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":140,"threads_started":1}
I20260929 16:35:10.288450 12371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:10.289969 12371 log.cc:824] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:10.290406 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:10.292287 12371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d: No bootstrap required, opened a new log
I20260929 16:35:10.296350 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:10.292452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:60585 (local address 127.0.0.1:39981)
0929 16:35:10.292574 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:10.292578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:10.292724 (+ 146us) server_negotiation.cc:407] Connection header received
0929 16:35:10.292808 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:10.292811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:10.292854 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:10.292928 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:10.293142 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.293880 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.294710 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:10.295031 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:10.295138 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:10.295944 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:10.295949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:10.295952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:10.295964 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:10.295989 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:10.296001 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:10.296097 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:10.296192 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 16:35:10.296224 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:35:10.296798 12371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7069cf0cc34f43328ec773c9f689b53d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39981 } }
I20260929 16:35:10.297049 12371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:10.297130 12371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7069cf0cc34f43328ec773c9f689b53d, State: Initialized, Role: FOLLOWER
I20260929 16:35:10.297565 12371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7069cf0cc34f43328ec773c9f689b53d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39981 } }
I20260929 16:35:10.297685 12371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:10.297724 12371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:10.297791 12371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:10.298544 12371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7069cf0cc34f43328ec773c9f689b53d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39981 } }
I20260929 16:35:10.298815 12371 leader_election.cc:302] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7069cf0cc34f43328ec773c9f689b53d; no voters:
I20260929 16:35:10.299041 12371 leader_election.cc:288] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:10.299158 12381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:10.299391 12381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [term 1 LEADER]: Becoming Leader. State: Replica: 7069cf0cc34f43328ec773c9f689b53d, State: Running, Role: LEADER
I20260929 16:35:10.299721 12381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7069cf0cc34f43328ec773c9f689b53d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39981 } }
I20260929 16:35:10.300145 12371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:10.301220 12383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7069cf0cc34f43328ec773c9f689b53d. Latest consensus state: current_term: 1 leader_uuid: "7069cf0cc34f43328ec773c9f689b53d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7069cf0cc34f43328ec773c9f689b53d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39981 } } }
I20260929 16:35:10.301373 12383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:10.301445 12382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7069cf0cc34f43328ec773c9f689b53d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7069cf0cc34f43328ec773c9f689b53d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39981 } } }
I20260929 16:35:10.301534 12382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d [sys.catalog]: This master's current role is: LEADER
W20260929 16:35:10.304915 12396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:35:10.305003 12396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:35:10.305083 12390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:10.305795 12390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:10.310772 12390 catalog_manager.cc:1347] Generated new cluster ID: 571e82df5f064c3094a8af238f5ef536
I20260929 16:35:10.310832 12390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:10.320307 12390 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:10.321419 12390 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:10.332072 12390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7069cf0cc34f43328ec773c9f689b53d: Generated new TSK 0
I20260929 16:35:10.332652 12390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:10.339864 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 12303
W20260929 16:35:10.345199 12288 connection.cc:535] client connection to 127.0.0.1:39981 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39981): Connection reset by peer (error 104)
2026-09-29T16:35:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:10 dist-test-slave-mlmz krb5kdc[12402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:10 dist-test-slave-mlmz krb5kdc[12402](info): set up 2 sockets
Sep 29 16:35:10 dist-test-slave-mlmz krb5kdc[12402](info): commencing operation
krb5kdc: starting...
W20260929 16:35:11.361493 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:11 dist-test-slave-mlmz krb5kdc[12402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:35:11.480255 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:11.585040 12415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:11.585316 12415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:11.585422 12415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:11.589269 12415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:11.589349 12415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:11.589395 12415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:11.589452 12415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:11.589491 12415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:11.594993 12415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:11.595868 12415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:11.596762 12415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:11.604899 12421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:11.604904 12423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:11.604904 12420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:11.605149 12415 server_base.cc:1034] running on GCE node
I20260929 16:35:11.605811 12415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:11.606863 12415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:11.608063 12415 hybrid_clock.cc:648] HybridClock initialized: now 1790699711608050 us; error 34 us; skew 500 ppm
Sep 29 16:35:11 dist-test-slave-mlmz krb5kdc[12402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:11.614825 12415 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:35:11.615737 12415 webserver.cc:460] Webserver started at http://127.0.0.1:45487/ using document root <none> and password file <none>
I20260929 16:35:11.616317 12415 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:11.616415 12415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:11.616650 12415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:11.618762 12415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "0b1ca92d308c4792916c26d7bafb9dee"
format_stamp: "Formatted at 2026-09-29 16:35:11 on dist-test-slave-mlmz"
server_key: "76df58059952d6b003daaf4dd32fda79"
server_key_iv: "58dde042ef9df043dbee39079456edfc"
server_key_version: "encryptionkey@0"
I20260929 16:35:11.619369 12415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "0b1ca92d308c4792916c26d7bafb9dee"
format_stamp: "Formatted at 2026-09-29 16:35:11 on dist-test-slave-mlmz"
server_key: "76df58059952d6b003daaf4dd32fda79"
server_key_iv: "58dde042ef9df043dbee39079456edfc"
server_key_version: "encryptionkey@0"
I20260929 16:35:11.623126 12415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:35:11.625635 12430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:11.626807 12415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:35:11.626955 12415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "0b1ca92d308c4792916c26d7bafb9dee"
format_stamp: "Formatted at 2026-09-29 16:35:11 on dist-test-slave-mlmz"
server_key: "76df58059952d6b003daaf4dd32fda79"
server_key_iv: "58dde042ef9df043dbee39079456edfc"
server_key_version: "encryptionkey@0"
I20260929 16:35:11.627100 12415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:11.631414 12415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:11.634205 12415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:11.634393 12415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:11.640987 12415 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35497
I20260929 16:35:11.641019 12482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35497 every 8 connection(s)
I20260929 16:35:11.641855 12415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:11.644351 12483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:11.645246 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 12415
I20260929 16:35:11.645395 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:11.645725 23728 external_mini_cluster.cc:1428] Setting key 5cf5722fb378fc9a29f08567f905f053
I20260929 16:35:11.650327 12483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee: Bootstrap starting.
I20260929 16:35:11.650534 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.647008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:55191 (local address 127.0.0.1:35497)
0929 16:35:11.647413 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.647427 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.647467 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:35:11.648062 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.648078 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.648327 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.648649 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:11.649392 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:11.649398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:11.649419 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:11.649429 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:11.649485 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:11.649503 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:11.649646 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.649715 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.649859 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260929 16:35:11.652659 12483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:11.653299 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:11.653548 12483 log.cc:824] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:11.655330 12483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee: No bootstrap required, opened a new log
I20260929 16:35:11.656966 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:11.655272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:50717 (local address 127.0.0.1:35497)
0929 16:35:11.655394 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:11.655399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:11.655661 (+ 262us) server_negotiation.cc:407] Connection header received
0929 16:35:11.655751 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:11.655753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:11.655779 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:11.655842 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:11.656534 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:11.656537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:11.656539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:11.656561 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:11.656585 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:11.656596 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:11.656675 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:11.656774 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 16:35:11.656804 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:35:11.659461 12483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1ca92d308c4792916c26d7bafb9dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } }
I20260929 16:35:11.659701 12483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:11.659763 12483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b1ca92d308c4792916c26d7bafb9dee, State: Initialized, Role: FOLLOWER
I20260929 16:35:11.660138 12483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1ca92d308c4792916c26d7bafb9dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } }
I20260929 16:35:11.660282 12483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:11.660346 12483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:11.660432 12483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:11.661242 12483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1ca92d308c4792916c26d7bafb9dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } }
I20260929 16:35:11.661585 12483 leader_election.cc:302] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b1ca92d308c4792916c26d7bafb9dee; no voters:
I20260929 16:35:11.661896 12483 leader_election.cc:288] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:11.662026 12493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:11.662294 12493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [term 1 LEADER]: Becoming Leader. State: Replica: 0b1ca92d308c4792916c26d7bafb9dee, State: Running, Role: LEADER
I20260929 16:35:11.662626 12493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1ca92d308c4792916c26d7bafb9dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } }
I20260929 16:35:11.663081 12483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:11.664294 12494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b1ca92d308c4792916c26d7bafb9dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1ca92d308c4792916c26d7bafb9dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } } }
I20260929 16:35:11.664292 12495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b1ca92d308c4792916c26d7bafb9dee. Latest consensus state: current_term: 1 leader_uuid: "0b1ca92d308c4792916c26d7bafb9dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1ca92d308c4792916c26d7bafb9dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } } }
I20260929 16:35:11.664491 12495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:11.664491 12494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:11.664857 12503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:11.667402 12503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:11.672439 12503 catalog_manager.cc:1347] Generated new cluster ID: 0ac47c5a0c3d412e9a84785fd60ca33e
I20260929 16:35:11.672502 12503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:11.698760 12503 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:11.699084 12503 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:11.720157 12503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b1ca92d308c4792916c26d7bafb9dee: Generated new TSK 0
I20260929 16:35:11.720948 12503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:11.736912 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 12415
2026-09-29T16:35:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:11 dist-test-slave-mlmz krb5kdc[12514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:11 dist-test-slave-mlmz krb5kdc[12514](info): set up 2 sockets
Sep 29 16:35:11 dist-test-slave-mlmz krb5kdc[12514](info): commencing operation
krb5kdc: starting...
W20260929 16:35:12.773463 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:12 dist-test-slave-mlmz krb5kdc[12514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:35:12.887925 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:12.991093 12527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:12.991353 12527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:12.991425 12527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:12.995278 12527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:12.995359 12527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:12.995400 12527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:12.995458 12527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:12.995499 12527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:13.000415 12527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:13.001330 12527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:13.002246 12527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:13.010042 12532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:13.010025 12535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:13.010025 12533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:13.010561 12527 server_base.cc:1034] running on GCE node
I20260929 16:35:13.011055 12527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:13.012039 12527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:13.013250 12527 hybrid_clock.cc:648] HybridClock initialized: now 1790699713013231 us; error 28 us; skew 500 ppm
Sep 29 16:35:13 dist-test-slave-mlmz krb5kdc[12514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:13.015837 12527 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:35:13.016680 12527 webserver.cc:460] Webserver started at http://127.0.0.1:36105/ using document root <none> and password file <none>
I20260929 16:35:13.017196 12527 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:13.017306 12527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:13.017524 12527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:13.019362 12527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "f76bf0fd3d66415e9b736d7698a78d53"
format_stamp: "Formatted at 2026-09-29 16:35:13 on dist-test-slave-mlmz"
server_key: "6f2f1a3e2454a30aa1594d339abc0790"
server_key_iv: "b2ac95abac9880103e28551cbdc44cd2"
server_key_version: "encryptionkey@0"
I20260929 16:35:13.019877 12527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "f76bf0fd3d66415e9b736d7698a78d53"
format_stamp: "Formatted at 2026-09-29 16:35:13 on dist-test-slave-mlmz"
server_key: "6f2f1a3e2454a30aa1594d339abc0790"
server_key_iv: "b2ac95abac9880103e28551cbdc44cd2"
server_key_version: "encryptionkey@0"
I20260929 16:35:13.023265 12527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 16:35:13.025477 12542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:13.026543 12527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:13.026674 12527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "f76bf0fd3d66415e9b736d7698a78d53"
format_stamp: "Formatted at 2026-09-29 16:35:13 on dist-test-slave-mlmz"
server_key: "6f2f1a3e2454a30aa1594d339abc0790"
server_key_iv: "b2ac95abac9880103e28551cbdc44cd2"
server_key_version: "encryptionkey@0"
I20260929 16:35:13.026774 12527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:13.043170 12527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:13.045727 12527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:13.045913 12527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:13.052779 12527 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38469
I20260929 16:35:13.052805 12594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38469 every 8 connection(s)
I20260929 16:35:13.053689 12527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:13.056154 12595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:13.060901 12595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53: Bootstrap starting.
I20260929 16:35:13.062888 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 12527
I20260929 16:35:13.063026 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:13.063261 12595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:13.063336 23728 external_mini_cluster.cc:1428] Setting key 450530140e7e89208b736719b0962dba
I20260929 16:35:13.064072 12595 log.cc:824] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:13.066190 12595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53: No bootstrap required, opened a new log
I20260929 16:35:13.070325 12595 raft_consensus.cc:348] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76bf0fd3d66415e9b736d7698a78d53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38469 } }
I20260929 16:35:13.070628 12595 raft_consensus.cc:374] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:13.070705 12595 raft_consensus.cc:729] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76bf0fd3d66415e9b736d7698a78d53, State: Initialized, Role: FOLLOWER
I20260929 16:35:13.071132 12595 consensus_queue.cc:260] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76bf0fd3d66415e9b736d7698a78d53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38469 } }
I20260929 16:35:13.071272 12595 raft_consensus.cc:388] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:13.071336 12595 raft_consensus.cc:482] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:13.071442 12595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:13.072225 12595 raft_consensus.cc:504] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76bf0fd3d66415e9b736d7698a78d53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38469 } }
Sep 29 16:35:13 dist-test-slave-mlmz krb5kdc[12514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:35:13.072544 12595 leader_election.cc:302] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f76bf0fd3d66415e9b736d7698a78d53; no voters:
I20260929 16:35:13.072883 12595 leader_election.cc:288] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:13.072957 12600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:13.073170 12600 raft_consensus.cc:686] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [term 1 LEADER]: Becoming Leader. State: Replica: f76bf0fd3d66415e9b736d7698a78d53, State: Running, Role: LEADER
I20260929 16:35:13.073503 12600 consensus_queue.cc:237] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76bf0fd3d66415e9b736d7698a78d53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38469 } }
I20260929 16:35:13.074184 12595 sys_catalog.cc:564] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:13.074549 12600 sys_catalog.cc:455] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f76bf0fd3d66415e9b736d7698a78d53. Latest consensus state: current_term: 1 leader_uuid: "f76bf0fd3d66415e9b736d7698a78d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76bf0fd3d66415e9b736d7698a78d53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38469 } } }
I20260929 16:35:13.074671 12600 sys_catalog.cc:458] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:13.075111 12601 sys_catalog.cc:455] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f76bf0fd3d66415e9b736d7698a78d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76bf0fd3d66415e9b736d7698a78d53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38469 } } }
I20260929 16:35:13.075212 12601 sys_catalog.cc:458] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:13.075462 12604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:13.078679 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:13.064376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:40711 (local address 127.0.0.1:38469)
0929 16:35:13.064884 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:13.064916 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:13.064959 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:35:13.065767 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:13.065787 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:13.066180 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:13.066538 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:13.067125 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:13.068582 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:13.069477 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:13.070167 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:13.070327 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:13.072892 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:13.072911 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:13.072928 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:13.072961 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:13.075088 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:13.075750 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:13.075756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:13.075761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:13.075920 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:13.076367 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:13.076370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:13.076372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:13.076834 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:13.077731 (+ 897us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:13.077895 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 16:35:13.078054 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":107,"threads_started":1}
I20260929 16:35:13.078874 12604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:13.081485 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:13.084649 12604 catalog_manager.cc:1347] Generated new cluster ID: 208ff46a57ed45eb87700be2d58694bb
I20260929 16:35:13.084755 12604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:13.091372 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:13.083672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:54781 (local address 127.0.0.1:38469)
0929 16:35:13.083770 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:13.083774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:13.084011 (+ 237us) server_negotiation.cc:407] Connection header received
0929 16:35:13.084177 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:13.084180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:13.084223 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:13.084270 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:13.085138 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:13.085716 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:13.086486 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:13.086749 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:13.086857 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:13.088171 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:13.088175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:13.088177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:13.088200 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:13.089651 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:13.090121 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:13.090125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:13.090127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:13.090204 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:13.090530 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:13.090533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:13.090535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:13.090695 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:13.090846 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:13.091114 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:35:13.091200 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:35:13.103871 12604 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:13.104831 12604 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:13.117146 12604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f76bf0fd3d66415e9b736d7698a78d53: Generated new TSK 0
I20260929 16:35:13.117772 12604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:13.122030 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 12527
W20260929 16:35:13.127247 12512 connection.cc:535] client connection to 127.0.0.1:38469 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38469): Connection reset by peer (error 104)
2026-09-29T16:35:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:13 dist-test-slave-mlmz krb5kdc[12626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:13 dist-test-slave-mlmz krb5kdc[12626](info): set up 2 sockets
Sep 29 16:35:13 dist-test-slave-mlmz krb5kdc[12626](info): commencing operation
krb5kdc: starting...
W20260929 16:35:14.153515 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:14 dist-test-slave-mlmz krb5kdc[12626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:35:14.266582 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:14.369765 12639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:14.369983 12639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:14.370023 12639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:14.373915 12639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:14.373965 12639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:14.373984 12639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:14.374001 12639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:14.374017 12639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:14.378930 12639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:14.379827 12639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:14.380728 12639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:14.388460 12645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:14.388705 12639 server_base.cc:1034] running on GCE node
W20260929 16:35:14.388489 12647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:14.388515 12644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:14.389374 12639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:14.390359 12639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:14.391551 12639 hybrid_clock.cc:648] HybridClock initialized: now 1790699714391532 us; error 30 us; skew 500 ppm
Sep 29 16:35:14 dist-test-slave-mlmz krb5kdc[12626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:14.394212 12639 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:35:14.395013 12639 webserver.cc:460] Webserver started at http://127.0.0.1:43361/ using document root <none> and password file <none>
I20260929 16:35:14.395514 12639 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:14.395596 12639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:14.395797 12639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:14.397673 12639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "bfcd713b78134491ab6ffa235f0a6896"
format_stamp: "Formatted at 2026-09-29 16:35:14 on dist-test-slave-mlmz"
server_key: "02e81700c4b7309957ddda82c2a4f51c"
server_key_iv: "396f29fcee4436a88b81dad0cfbe231e"
server_key_version: "encryptionkey@0"
I20260929 16:35:14.398191 12639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "bfcd713b78134491ab6ffa235f0a6896"
format_stamp: "Formatted at 2026-09-29 16:35:14 on dist-test-slave-mlmz"
server_key: "02e81700c4b7309957ddda82c2a4f51c"
server_key_iv: "396f29fcee4436a88b81dad0cfbe231e"
server_key_version: "encryptionkey@0"
I20260929 16:35:14.401559 12639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:14.403788 12654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:14.404929 12639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:14.405043 12639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "bfcd713b78134491ab6ffa235f0a6896"
format_stamp: "Formatted at 2026-09-29 16:35:14 on dist-test-slave-mlmz"
server_key: "02e81700c4b7309957ddda82c2a4f51c"
server_key_iv: "396f29fcee4436a88b81dad0cfbe231e"
server_key_version: "encryptionkey@0"
I20260929 16:35:14.405136 12639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:14.429702 12639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:14.432605 12639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:14.432803 12639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:14.439294 12639 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46595
I20260929 16:35:14.439324 12706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46595 every 8 connection(s)
I20260929 16:35:14.440133 12639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:14.441591 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 12639
I20260929 16:35:14.441713 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:14.441973 23728 external_mini_cluster.cc:1428] Setting key 28c23d2aee9d1ab37df7f0a8e88edf36
I20260929 16:35:14.442859 12707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:14.448055 12707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896: Bootstrap starting.
I20260929 16:35:14.450062 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:14.443212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:34649 (local address 127.0.0.1:46595)
0929 16:35:14.443620 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:14.443633 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:14.443664 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:35:14.444268 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:14.444282 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:14.444588 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:14.444899 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:14.445321 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:14.446259 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:14.447112 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:14.447566 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:14.447719 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:14.448563 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:14.448568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:14.448580 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:14.448917 (+ 337us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:14.448964 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:14.448985 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:14.449126 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:14.449310 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 16:35:14.449436 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":147,"threads_started":1}
I20260929 16:35:14.450171 12707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:14.450825 12707 log.cc:824] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:14.452795 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:14.452939 12707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896: No bootstrap required, opened a new log
I20260929 16:35:14.457089 12707 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfcd713b78134491ab6ffa235f0a6896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260929 16:35:14.457333 12707 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:14.457412 12707 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfcd713b78134491ab6ffa235f0a6896, State: Initialized, Role: FOLLOWER
I20260929 16:35:14.457836 12707 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfcd713b78134491ab6ffa235f0a6896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260929 16:35:14.458019 12707 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:14.458091 12707 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:14.458199 12707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:14.458684 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:14.454662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:36825 (local address 127.0.0.1:46595)
0929 16:35:14.454757 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:14.454760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:14.455017 (+ 257us) server_negotiation.cc:407] Connection header received
0929 16:35:14.455160 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:14.455162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:14.455194 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:14.455250 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:14.455532 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:14.455976 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:14.457058 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:14.457308 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:14.457425 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:14.458185 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:14.458189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:14.458192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:14.458203 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:14.458227 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:14.458238 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:14.458344 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:14.458511 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 16:35:14.458537 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 16:35:14.459215 12707 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfcd713b78134491ab6ffa235f0a6896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260929 16:35:14.459545 12707 leader_election.cc:302] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfcd713b78134491ab6ffa235f0a6896; no voters:
I20260929 16:35:14.459838 12707 leader_election.cc:288] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:14.459949 12717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:14.460160 12717 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [term 1 LEADER]: Becoming Leader. State: Replica: bfcd713b78134491ab6ffa235f0a6896, State: Running, Role: LEADER
I20260929 16:35:14.460496 12717 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfcd713b78134491ab6ffa235f0a6896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260929 16:35:14.460988 12707 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:14.462127 12718 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfcd713b78134491ab6ffa235f0a6896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfcd713b78134491ab6ffa235f0a6896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } } }
I20260929 16:35:14.462160 12719 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfcd713b78134491ab6ffa235f0a6896. Latest consensus state: current_term: 1 leader_uuid: "bfcd713b78134491ab6ffa235f0a6896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfcd713b78134491ab6ffa235f0a6896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } } }
I20260929 16:35:14.462255 12718 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:14.462255 12719 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:14.463119 12726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:14.465816 12726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:14.470255 12726 catalog_manager.cc:1347] Generated new cluster ID: 197b795645524976ac2b651ab26b33ad
I20260929 16:35:14.470314 12726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:14.476001 12726 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:14.477072 12726 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:14.488023 12726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfcd713b78134491ab6ffa235f0a6896: Generated new TSK 0
I20260929 16:35:14.488700 12726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:14.494199 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 12639
W20260929 16:35:14.498687 12624 connection.cc:535] client connection to 127.0.0.1:46595 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46595): Connection reset by peer (error 104)
2026-09-29T16:35:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:14 dist-test-slave-mlmz krb5kdc[12738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:14 dist-test-slave-mlmz krb5kdc[12738](info): set up 2 sockets
Sep 29 16:35:14 dist-test-slave-mlmz krb5kdc[12738](info): commencing operation
krb5kdc: starting...
W20260929 16:35:15.541437 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:15 dist-test-slave-mlmz krb5kdc[12738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:35:15.652657 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:15.754673 12751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:15.754894 12751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:15.754933 12751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:15.758888 12751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:15.758934 12751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:15.758952 12751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:15.758968 12751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:15.758982 12751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:15.763779 12751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45345
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:15.764590 12751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:15.765441 12751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:15.773121 12756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:15.773109 12759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:15.773109 12757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:15.773456 12751 server_base.cc:1034] running on GCE node
I20260929 16:35:15.773968 12751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:15.774928 12751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:15.776099 12751 hybrid_clock.cc:648] HybridClock initialized: now 1790699715776073 us; error 40 us; skew 500 ppm
Sep 29 16:35:15 dist-test-slave-mlmz krb5kdc[12738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:15.778715 12751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:35:15.779469 12751 webserver.cc:460] Webserver started at http://127.0.0.1:42391/ using document root <none> and password file <none>
I20260929 16:35:15.779919 12751 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:15.779973 12751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:15.780117 12751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:15.781939 12751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "6875f40779aa4f4aab6a889fcedef5a3"
format_stamp: "Formatted at 2026-09-29 16:35:15 on dist-test-slave-mlmz"
server_key: "029b38c4139b63efca69e9fbdaadf565"
server_key_iv: "5761376762e259bbbfe2c2c6450766ac"
server_key_version: "encryptionkey@0"
I20260929 16:35:15.782403 12751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "6875f40779aa4f4aab6a889fcedef5a3"
format_stamp: "Formatted at 2026-09-29 16:35:15 on dist-test-slave-mlmz"
server_key: "029b38c4139b63efca69e9fbdaadf565"
server_key_iv: "5761376762e259bbbfe2c2c6450766ac"
server_key_version: "encryptionkey@0"
I20260929 16:35:15.785672 12751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:15.787812 12766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:15.788972 12751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:15.789064 12751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "6875f40779aa4f4aab6a889fcedef5a3"
format_stamp: "Formatted at 2026-09-29 16:35:15 on dist-test-slave-mlmz"
server_key: "029b38c4139b63efca69e9fbdaadf565"
server_key_iv: "5761376762e259bbbfe2c2c6450766ac"
server_key_version: "encryptionkey@0"
I20260929 16:35:15.789139 12751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:15.793313 12751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:15.796408 12751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:15.796548 12751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:15.803112 12751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45345
I20260929 16:35:15.803134 12818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45345 every 8 connection(s)
I20260929 16:35:15.803922 12751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:15.806380 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:15.807397 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 12751
I20260929 16:35:15.807518 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:15.807780 23728 external_mini_cluster.cc:1428] Setting key 28b112ee39b149c5e043c3d1f087df4f
I20260929 16:35:15.812618 12819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3: Bootstrap starting.
I20260929 16:35:15.812798 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:15.809011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:57415 (local address 127.0.0.1:45345)
0929 16:35:15.809407 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:15.809427 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:15.809468 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:35:15.810242 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:15.810263 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:15.810586 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:15.810925 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:15.811509 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:15.811515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:15.811528 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:15.811539 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:15.811594 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:15.811619 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:15.811777 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:15.811855 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 16:35:15.811993 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":89,"threads_started":1}
I20260929 16:35:15.814990 12819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:15.815227 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:15.815701 12819 log.cc:824] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:15.817608 12819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3: No bootstrap required, opened a new log
I20260929 16:35:15.818737 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:15.817200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:33343 (local address 127.0.0.1:45345)
0929 16:35:15.817329 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:15.817333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:15.817486 (+ 153us) server_negotiation.cc:407] Connection header received
0929 16:35:15.817631 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:15.817634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:15.817663 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:15.817746 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:15.818360 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:15.818362 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:15.818364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:15.818374 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:15.818394 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:15.818403 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:15.818463 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:15.818576 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 16:35:15.818605 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 16:35:15.821504 12819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6875f40779aa4f4aab6a889fcedef5a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } }
I20260929 16:35:15.821724 12819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:15.821794 12819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6875f40779aa4f4aab6a889fcedef5a3, State: Initialized, Role: FOLLOWER
I20260929 16:35:15.822167 12819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6875f40779aa4f4aab6a889fcedef5a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } }
I20260929 16:35:15.822304 12819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:15.822378 12819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:15.822484 12819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:15.823280 12819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6875f40779aa4f4aab6a889fcedef5a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } }
I20260929 16:35:15.823597 12819 leader_election.cc:302] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6875f40779aa4f4aab6a889fcedef5a3; no voters:
I20260929 16:35:15.823874 12819 leader_election.cc:288] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:15.823961 12829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:15.824261 12829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [term 1 LEADER]: Becoming Leader. State: Replica: 6875f40779aa4f4aab6a889fcedef5a3, State: Running, Role: LEADER
I20260929 16:35:15.824589 12829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6875f40779aa4f4aab6a889fcedef5a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } }
I20260929 16:35:15.825105 12819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:15.826340 12831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6875f40779aa4f4aab6a889fcedef5a3. Latest consensus state: current_term: 1 leader_uuid: "6875f40779aa4f4aab6a889fcedef5a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6875f40779aa4f4aab6a889fcedef5a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } } }
I20260929 16:35:15.826340 12830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6875f40779aa4f4aab6a889fcedef5a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6875f40779aa4f4aab6a889fcedef5a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } } }
I20260929 16:35:15.826543 12831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:15.826552 12830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:15.826944 12843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:15.829674 12843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:15.834295 12843 catalog_manager.cc:1347] Generated new cluster ID: 41b76ff35d494427920aef3014df9cf0
I20260929 16:35:15.834383 12843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:15.848762 12843 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:15.849073 12843 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:15.856701 12843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6875f40779aa4f4aab6a889fcedef5a3: Generated new TSK 0
I20260929 16:35:15.857273 12843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:15.870335 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 12751
2026-09-29T16:35:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:15 dist-test-slave-mlmz krb5kdc[12850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:15 dist-test-slave-mlmz krb5kdc[12850](info): set up 2 sockets
Sep 29 16:35:15 dist-test-slave-mlmz krb5kdc[12850](info): commencing operation
krb5kdc: starting...
W20260929 16:35:16.893484 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:16 dist-test-slave-mlmz krb5kdc[12850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:35:17.009584 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:42579
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:42579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:17.113108 12863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:17.113348 12863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:17.113387 12863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:17.117234 12863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:17.117300 12863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:17.117318 12863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:17.117334 12863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:17.117388 12863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:17.122387 12863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:42579
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:42579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:17.123281 12863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:17.124194 12863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:17.132112 12868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:17.132133 12869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:17.132133 12871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:17.132185 12863 server_base.cc:1034] running on GCE node
I20260929 16:35:17.132838 12863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:17.133911 12863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:17.135119 12863 hybrid_clock.cc:648] HybridClock initialized: now 1790699717135105 us; error 27 us; skew 500 ppm
Sep 29 16:35:17 dist-test-slave-mlmz krb5kdc[12850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699717, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:17.142652 12863 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:35:17.143452 12863 webserver.cc:460] Webserver started at http://127.23.44.62:46245/ using document root <none> and password file <none>
I20260929 16:35:17.143983 12863 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:17.144101 12863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:17.144315 12863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:17.146310 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "1a8bbe859ac84f1da7673a0886de85be"
format_stamp: "Formatted at 2026-09-29 16:35:17 on dist-test-slave-mlmz"
server_key: "1a79a25ee1ee88a016189e0eee6267cb"
server_key_iv: "2c7f0749c1fa0c28e0c66c4b5d8ab632"
server_key_version: "encryptionkey@0"
I20260929 16:35:17.146847 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "1a8bbe859ac84f1da7673a0886de85be"
format_stamp: "Formatted at 2026-09-29 16:35:17 on dist-test-slave-mlmz"
server_key: "1a79a25ee1ee88a016189e0eee6267cb"
server_key_iv: "2c7f0749c1fa0c28e0c66c4b5d8ab632"
server_key_version: "encryptionkey@0"
I20260929 16:35:17.150277 12863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 16:35:17.152529 12878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:17.153648 12863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:35:17.153781 12863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "1a8bbe859ac84f1da7673a0886de85be"
format_stamp: "Formatted at 2026-09-29 16:35:17 on dist-test-slave-mlmz"
server_key: "1a79a25ee1ee88a016189e0eee6267cb"
server_key_iv: "2c7f0749c1fa0c28e0c66c4b5d8ab632"
server_key_version: "encryptionkey@0"
I20260929 16:35:17.153884 12863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:17.187713 12863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:17.191624 12863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:17.191847 12863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:17.200476 12863 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:42579
I20260929 16:35:17.200542 12930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:42579 every 8 connection(s)
I20260929 16:35:17.201457 12863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:17.204034 12931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:17.204826 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 12863
I20260929 16:35:17.204974 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:17.205343 23728 external_mini_cluster.cc:1428] Setting key 30538874cbc4a28a3c32b424c4484de1
I20260929 16:35:17.210258 12931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be: Bootstrap starting.
I20260929 16:35:17.212628 12931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:17.213351 12931 log.cc:824] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:35:17 dist-test-slave-mlmz krb5kdc[12850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:35:17.215226 12931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be: No bootstrap required, opened a new log
I20260929 16:35:17.219827 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:17.206670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60215 (local address 127.23.44.62:42579)
0929 16:35:17.207056 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:17.207077 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:17.207122 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:35:17.207901 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:17.207923 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:17.208315 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:17.208655 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:17.209327 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:17.210714 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:17.211586 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:17.212246 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:17.212398 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:17.214866 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:17.214872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:17.214897 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:17.214930 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:17.216824 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:17.217340 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:17.217345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:17.217350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:17.217501 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:17.217819 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:17.217823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:17.217826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:17.218300 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:17.218498 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:17.218720 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 16:35:17.218941 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":96,"threads_started":1}
I20260929 16:35:17.219806 12931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8bbe859ac84f1da7673a0886de85be" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42579 } }
I20260929 16:35:17.220091 12931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:17.220153 12931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8bbe859ac84f1da7673a0886de85be, State: Initialized, Role: FOLLOWER
I20260929 16:35:17.220539 12931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8bbe859ac84f1da7673a0886de85be" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42579 } }
I20260929 16:35:17.220677 12931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:17.220739 12931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:17.220829 12931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:17.221721 12931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8bbe859ac84f1da7673a0886de85be" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42579 } }
I20260929 16:35:17.222056 12931 leader_election.cc:302] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a8bbe859ac84f1da7673a0886de85be; no voters:
I20260929 16:35:17.222383 12931 leader_election.cc:288] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:17.222440 12936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:17.222644 12936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [term 1 LEADER]: Becoming Leader. State: Replica: 1a8bbe859ac84f1da7673a0886de85be, State: Running, Role: LEADER
I20260929 16:35:17.222836 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:17.222971 12936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8bbe859ac84f1da7673a0886de85be" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42579 } }
I20260929 16:35:17.223580 12931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:17.224464 12937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a8bbe859ac84f1da7673a0886de85be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8bbe859ac84f1da7673a0886de85be" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42579 } } }
I20260929 16:35:17.224589 12937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:17.224936 12938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a8bbe859ac84f1da7673a0886de85be. Latest consensus state: current_term: 1 leader_uuid: "1a8bbe859ac84f1da7673a0886de85be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8bbe859ac84f1da7673a0886de85be" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42579 } } }
I20260929 16:35:17.225036 12938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:17.225378 12949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:17.228330 12949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:17.233655 12949 catalog_manager.cc:1347] Generated new cluster ID: 342f2ad237fe43d9b4aa6c737e82622e
I20260929 16:35:17.233745 12949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:17.233901 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:17.224886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34957 (local address 127.23.44.62:42579)
0929 16:35:17.225108 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:17.225114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:17.225184 (+ 70us) server_negotiation.cc:407] Connection header received
0929 16:35:17.225381 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:17.225384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:17.225429 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:17.225480 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:17.226389 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:17.227101 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:17.228202 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:17.228525 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:17.228630 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:17.230235 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:17.230238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:17.230240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:17.230259 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:17.231722 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:17.232534 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:17.232538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:17.232539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:17.232622 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:17.232999 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:17.233003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:17.233005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:17.233201 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:17.233405 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:17.233646 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 16:35:17.233740 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260929 16:35:17.239609 12949 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:17.240685 12949 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:17.249100 12949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a8bbe859ac84f1da7673a0886de85be: Generated new TSK 0
I20260929 16:35:17.249809 12949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:17.254937 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 12863
W20260929 16:35:17.260140 12848 connection.cc:535] client connection to 127.23.44.62:42579 recv error: Network error: failed to read from TLS socket (remote: 127.23.44.62:42579): Connection reset by peer (error 104)
2026-09-29T16:35:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:17 dist-test-slave-mlmz krb5kdc[12962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:17 dist-test-slave-mlmz krb5kdc[12962](info): set up 2 sockets
Sep 29 16:35:17 dist-test-slave-mlmz krb5kdc[12962](info): commencing operation
krb5kdc: starting...
W20260929 16:35:18.289294 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:18 dist-test-slave-mlmz krb5kdc[12962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:35:18.401298 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:37545
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:34151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:18.502842 12975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:18.503049 12975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:18.503087 12975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:18.506946 12975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:18.506991 12975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:18.507010 12975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:18.507026 12975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:18.507040 12975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:18.511811 12975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:34151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:37545
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:37545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:18.512604 12975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:18.513434 12975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:18.521039 12980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:18.521133 12975 server_base.cc:1034] running on GCE node
W20260929 16:35:18.521066 12983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:18.521283 12981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:18.521765 12975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:18.522701 12975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:18.523861 12975 hybrid_clock.cc:648] HybridClock initialized: now 1790699718523846 us; error 28 us; skew 500 ppm
Sep 29 16:35:18 dist-test-slave-mlmz krb5kdc[12962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699718, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:18.526448 12975 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:35:18.527285 12975 webserver.cc:460] Webserver started at http://127.23.44.62:46083/ using document root <none> and password file <none>
I20260929 16:35:18.527750 12975 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:18.527806 12975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:18.527949 12975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:18.529776 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "472495be007b450686ba7b6d32b0ff0d"
format_stamp: "Formatted at 2026-09-29 16:35:18 on dist-test-slave-mlmz"
server_key: "7e2d9dc898e84d3f5063d89486d076c3"
server_key_iv: "27fbc0e77a71bc5e5a557cb59f321f94"
server_key_version: "encryptionkey@0"
I20260929 16:35:18.530233 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "472495be007b450686ba7b6d32b0ff0d"
format_stamp: "Formatted at 2026-09-29 16:35:18 on dist-test-slave-mlmz"
server_key: "7e2d9dc898e84d3f5063d89486d076c3"
server_key_iv: "27fbc0e77a71bc5e5a557cb59f321f94"
server_key_version: "encryptionkey@0"
I20260929 16:35:18.533499 12975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:18.535624 12990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:18.536734 12975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:18.536854 12975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "472495be007b450686ba7b6d32b0ff0d"
format_stamp: "Formatted at 2026-09-29 16:35:18 on dist-test-slave-mlmz"
server_key: "7e2d9dc898e84d3f5063d89486d076c3"
server_key_iv: "27fbc0e77a71bc5e5a557cb59f321f94"
server_key_version: "encryptionkey@0"
I20260929 16:35:18.536945 12975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:18.580845 12975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:18.583966 12975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:18.584162 12975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:18.591048 12975 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:37545
I20260929 16:35:18.591068 13042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:37545 every 8 connection(s)
I20260929 16:35:18.591933 12975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:18.594498 13043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:18.596411 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 12975
I20260929 16:35:18.596542 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:18.596807 23728 external_mini_cluster.cc:1428] Setting key 5407b7e2b2c267157a49f2beacfa5ce9
I20260929 16:35:18.600216 13043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d: Bootstrap starting.
I20260929 16:35:18.602679 13043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:18.603609 13043 log.cc:824] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:18.605489 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:18.598099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50781 (local address 127.23.44.62:37545)
0929 16:35:18.598463 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:18.598478 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:18.598509 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:35:18.599298 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:18.599314 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:18.599627 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:18.599877 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:18.600207 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:18.601159 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:18.602326 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:18.602759 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:18.602904 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:18.603776 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:18.603781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:18.603792 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:18.604244 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:18.604307 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:18.604337 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:18.604443 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:18.604658 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 16:35:18.604789 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":117,"threads_started":1}
I20260929 16:35:18.605558 13043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d: No bootstrap required, opened a new log
I20260929 16:35:18.607971 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:18.608980 13043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472495be007b450686ba7b6d32b0ff0d" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 37545 } }
I20260929 16:35:18.609248 13043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:18.609351 13043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472495be007b450686ba7b6d32b0ff0d, State: Initialized, Role: FOLLOWER
I20260929 16:35:18.609786 13043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472495be007b450686ba7b6d32b0ff0d" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 37545 } }
I20260929 16:35:18.609959 13043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:18.610080 13043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:18.610190 13043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:18.610977 13043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472495be007b450686ba7b6d32b0ff0d" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 37545 } }
I20260929 16:35:18.611296 13043 leader_election.cc:302] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 472495be007b450686ba7b6d32b0ff0d; no voters:
I20260929 16:35:18.611593 13043 leader_election.cc:288] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:18.611688 13053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:18.611867 13053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [term 1 LEADER]: Becoming Leader. State: Replica: 472495be007b450686ba7b6d32b0ff0d, State: Running, Role: LEADER
I20260929 16:35:18.612191 13053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472495be007b450686ba7b6d32b0ff0d" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 37545 } }
I20260929 16:35:18.612753 13043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:18.613324 13053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 472495be007b450686ba7b6d32b0ff0d. Latest consensus state: current_term: 1 leader_uuid: "472495be007b450686ba7b6d32b0ff0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472495be007b450686ba7b6d32b0ff0d" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 37545 } } }
I20260929 16:35:18.613487 13053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:18.613962 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:18.609898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44901 (local address 127.23.44.62:37545)
0929 16:35:18.610013 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:18.610016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:18.610250 (+ 234us) server_negotiation.cc:407] Connection header received
0929 16:35:18.610376 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:18.610379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:18.610415 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:18.610497 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:18.610744 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:18.611298 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:18.612278 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:18.612541 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:18.612649 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:18.613485 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:18.613488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:18.613491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:18.613501 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:18.613526 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:18.613540 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:18.613635 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:18.613799 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 16:35:18.613831 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:35:18.614791 13054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "472495be007b450686ba7b6d32b0ff0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472495be007b450686ba7b6d32b0ff0d" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 37545 } } }
I20260929 16:35:18.614882 13054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:18.614974 13058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:18.617915 13058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:18.622908 13058 catalog_manager.cc:1347] Generated new cluster ID: 4c8aad02c6c64b0694103eb85563ed48
I20260929 16:35:18.622969 13058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:18.633117 13058 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:18.633936 13058 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:18.638142 13058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 472495be007b450686ba7b6d32b0ff0d: Generated new TSK 0
I20260929 16:35:18.638695 13058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:18.643072 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 12975
W20260929 16:35:18.648325 12960 connection.cc:535] client connection to 127.23.44.62:37545 recv error: Network error: failed to read from TLS socket (remote: 127.23.44.62:37545): Connection reset by peer (error 104)
2026-09-29T16:35:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:18 dist-test-slave-mlmz krb5kdc[13074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:18 dist-test-slave-mlmz krb5kdc[13074](info): set up 2 sockets
Sep 29 16:35:18 dist-test-slave-mlmz krb5kdc[13074](info): commencing operation
krb5kdc: starting...
W20260929 16:35:19.669806 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:19 dist-test-slave-mlmz krb5kdc[13074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:35:19.787597 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:44995
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:37387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:44995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:19.890611 13087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:19.890844 13087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:19.890911 13087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:19.894877 13087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:19.894924 13087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:19.894943 13087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:19.894989 13087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:19.895025 13087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:19.899987 13087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:37387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:44995
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:44995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:19.900878 13087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:19.901816 13087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:19.910074 13093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:19.910076 13092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:19.910076 13095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:19.910245 13087 server_base.cc:1034] running on GCE node
I20260929 16:35:19.910835 13087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:19.911787 13087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:19.912964 13087 hybrid_clock.cc:648] HybridClock initialized: now 1790699719912943 us; error 48 us; skew 500 ppm
Sep 29 16:35:19 dist-test-slave-mlmz krb5kdc[13074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699719, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:19.915491 13087 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:35:19.916306 13087 webserver.cc:460] Webserver started at http://127.23.44.62:41781/ using document root <none> and password file <none>
I20260929 16:35:19.916810 13087 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:19.916893 13087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:19.917100 13087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:19.919008 13087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "5c3d69f2fa9140409f359c3129e470f1"
format_stamp: "Formatted at 2026-09-29 16:35:19 on dist-test-slave-mlmz"
server_key: "b04ea96b7aadd2ba5ee13edfb26eb5bc"
server_key_iv: "740c771b5ae5bfa116aa14cf0544620b"
server_key_version: "encryptionkey@0"
I20260929 16:35:19.919534 13087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "5c3d69f2fa9140409f359c3129e470f1"
format_stamp: "Formatted at 2026-09-29 16:35:19 on dist-test-slave-mlmz"
server_key: "b04ea96b7aadd2ba5ee13edfb26eb5bc"
server_key_iv: "740c771b5ae5bfa116aa14cf0544620b"
server_key_version: "encryptionkey@0"
I20260929 16:35:19.922873 13087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:35:19.925086 13102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:19.926196 13087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:35:19.926327 13087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "5c3d69f2fa9140409f359c3129e470f1"
format_stamp: "Formatted at 2026-09-29 16:35:19 on dist-test-slave-mlmz"
server_key: "b04ea96b7aadd2ba5ee13edfb26eb5bc"
server_key_iv: "740c771b5ae5bfa116aa14cf0544620b"
server_key_version: "encryptionkey@0"
I20260929 16:35:19.926427 13087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:19.930611 13087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:19.933348 13087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:19.933516 13087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:19.939939 13087 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:44995
I20260929 16:35:19.939952 13154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:44995 every 8 connection(s)
I20260929 16:35:19.940740 13087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:19.942456 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 13087
I20260929 16:35:19.942623 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:19.942889 23728 external_mini_cluster.cc:1428] Setting key 9a6483415087f89074cb14f598449f96
I20260929 16:35:19.943297 13155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:19.947943 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:19.944178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37981 (local address 127.23.44.62:44995)
0929 16:35:19.944556 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:19.944573 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:19.944616 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:35:19.945598 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:19.945615 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:19.945860 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:19.946121 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:19.946875 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:19.946880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:19.946890 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:19.946899 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:19.946944 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:19.946961 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:19.947075 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:19.947173 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 16:35:19.947326 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":138,"threads_started":1}
I20260929 16:35:19.949465 13155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1: Bootstrap starting.
I20260929 16:35:19.950493 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:19.951699 13155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:19.952404 13155 log.cc:824] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:19.953878 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:19.952321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44857 (local address 127.23.44.62:44995)
0929 16:35:19.952435 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:19.952439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:19.952623 (+ 184us) server_negotiation.cc:407] Connection header received
0929 16:35:19.952747 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:19.952750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:19.952786 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:19.952864 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:19.953496 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:19.953500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:19.953502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:19.953513 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:19.953536 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:19.953548 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:19.953635 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:19.953740 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 16:35:19.953775 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 16:35:19.954334 13155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1: No bootstrap required, opened a new log
I20260929 16:35:19.957723 13155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3d69f2fa9140409f359c3129e470f1" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44995 } }
I20260929 16:35:19.957909 13155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:19.957993 13155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3d69f2fa9140409f359c3129e470f1, State: Initialized, Role: FOLLOWER
I20260929 16:35:19.958407 13155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3d69f2fa9140409f359c3129e470f1" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44995 } }
I20260929 16:35:19.958539 13155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:19.958604 13155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:19.958693 13155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:19.959447 13155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3d69f2fa9140409f359c3129e470f1" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44995 } }
I20260929 16:35:19.959754 13155 leader_election.cc:302] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c3d69f2fa9140409f359c3129e470f1; no voters:
I20260929 16:35:19.960019 13155 leader_election.cc:288] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:19.960109 13165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:19.960341 13165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [term 1 LEADER]: Becoming Leader. State: Replica: 5c3d69f2fa9140409f359c3129e470f1, State: Running, Role: LEADER
I20260929 16:35:19.960660 13165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3d69f2fa9140409f359c3129e470f1" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44995 } }
I20260929 16:35:19.961124 13155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:19.962211 13167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c3d69f2fa9140409f359c3129e470f1. Latest consensus state: current_term: 1 leader_uuid: "5c3d69f2fa9140409f359c3129e470f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3d69f2fa9140409f359c3129e470f1" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44995 } } }
I20260929 16:35:19.962337 13167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:19.962630 13166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c3d69f2fa9140409f359c3129e470f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3d69f2fa9140409f359c3129e470f1" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 44995 } } }
I20260929 16:35:19.962702 13174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:19.962802 13166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:19.965791 13174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:19.970911 13174 catalog_manager.cc:1347] Generated new cluster ID: f7a5f801ab8f464982e75e9a7f3d2436
I20260929 16:35:19.970980 13174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:19.976585 13174 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:19.976908 13174 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:19.983093 13174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c3d69f2fa9140409f359c3129e470f1: Generated new TSK 0
I20260929 16:35:19.983660 13174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:19.988830 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 13087
2026-09-29T16:35:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:20 dist-test-slave-mlmz krb5kdc[13186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:20 dist-test-slave-mlmz krb5kdc[13186](info): set up 2 sockets
Sep 29 16:35:20 dist-test-slave-mlmz krb5kdc[13186](info): commencing operation
krb5kdc: starting...
W20260929 16:35:21.017342 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:21 dist-test-slave-mlmz krb5kdc[13186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:35:21.130218 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:46217
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:46217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:21.236166 13199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:21.236383 13199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:21.236423 13199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:21.240306 13199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:21.240353 13199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:21.240373 13199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:21.240389 13199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:21.240447 13199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:21.245446 13199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:46217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:46217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:21.246312 13199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:21.247205 13199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:21.255283 13204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:21.255299 13205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:21.255293 13207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:21.255492 13199 server_base.cc:1034] running on GCE node
I20260929 16:35:21.256065 13199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:21.256978 13199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:21.258138 13199 hybrid_clock.cc:648] HybridClock initialized: now 1790699721258095 us; error 55 us; skew 500 ppm
Sep 29 16:35:21 dist-test-slave-mlmz krb5kdc[13186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699721, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:21.260552 13199 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:35:21.261363 13199 webserver.cc:460] Webserver started at http://127.23.44.62:37621/ using document root <none> and password file <none>
I20260929 16:35:21.261862 13199 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:21.261920 13199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:21.262135 13199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:21.263976 13199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "1a2e2e0dfdaf4756b0cc381c1a98a13b"
format_stamp: "Formatted at 2026-09-29 16:35:21 on dist-test-slave-mlmz"
server_key: "d30ae0c1cc8b1f0ef8321c06e49ff0e3"
server_key_iv: "0b64455bfb1b63b414ba72261eec4204"
server_key_version: "encryptionkey@0"
I20260929 16:35:21.264487 13199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "1a2e2e0dfdaf4756b0cc381c1a98a13b"
format_stamp: "Formatted at 2026-09-29 16:35:21 on dist-test-slave-mlmz"
server_key: "d30ae0c1cc8b1f0ef8321c06e49ff0e3"
server_key_iv: "0b64455bfb1b63b414ba72261eec4204"
server_key_version: "encryptionkey@0"
I20260929 16:35:21.267973 13199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 16:35:21.270128 13214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:21.271198 13199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:21.271312 13199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "1a2e2e0dfdaf4756b0cc381c1a98a13b"
format_stamp: "Formatted at 2026-09-29 16:35:21 on dist-test-slave-mlmz"
server_key: "d30ae0c1cc8b1f0ef8321c06e49ff0e3"
server_key_iv: "0b64455bfb1b63b414ba72261eec4204"
server_key_version: "encryptionkey@0"
I20260929 16:35:21.271400 13199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:21.294080 13199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:21.296681 13199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:21.296860 13199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:21.304281 13199 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:46217
I20260929 16:35:21.304294 13266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:46217 every 8 connection(s)
I20260929 16:35:21.305128 13199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:21.305478 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 13199
I20260929 16:35:21.305644 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:21.305891 23728 external_mini_cluster.cc:1428] Setting key f920caebe6a13524d218362cceb5dac9
I20260929 16:35:21.307844 13267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:21.312815 13267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b: Bootstrap starting.
Sep 29 16:35:21 dist-test-slave-mlmz krb5kdc[13186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:35:21.314980 13267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:21.315799 13267 log.cc:824] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:21.317549 13267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b: No bootstrap required, opened a new log
I20260929 16:35:21.318615 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:21.307151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52167 (local address 127.23.44.62:46217)
0929 16:35:21.307544 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:21.307558 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:21.307589 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:35:21.308285 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:21.308300 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:21.308613 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:21.308916 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:21.309663 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:21.310594 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:21.311406 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:21.311846 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:21.312013 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:21.314365 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:21.314370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:21.314381 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:21.314406 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:21.316082 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:21.316483 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:21.316489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:21.316499 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:21.316606 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:21.316869 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:21.316873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:21.316875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:21.317228 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:21.317452 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:21.317696 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 16:35:21.317962 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":131,"threads_started":1}
I20260929 16:35:21.321045 13267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2e2e0dfdaf4756b0cc381c1a98a13b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46217 } }
I20260929 16:35:21.321290 13267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:21.321352 13267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2e2e0dfdaf4756b0cc381c1a98a13b, State: Initialized, Role: FOLLOWER
I20260929 16:35:21.321846 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:21.321800 13267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2e2e0dfdaf4756b0cc381c1a98a13b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46217 } }
I20260929 16:35:21.321949 13267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:21.322014 13267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:21.322103 13267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:21.322947 13267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2e2e0dfdaf4756b0cc381c1a98a13b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46217 } }
I20260929 16:35:21.323263 13267 leader_election.cc:302] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a2e2e0dfdaf4756b0cc381c1a98a13b; no voters:
I20260929 16:35:21.323588 13267 leader_election.cc:288] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:21.323747 13276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:21.324163 13276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [term 1 LEADER]: Becoming Leader. State: Replica: 1a2e2e0dfdaf4756b0cc381c1a98a13b, State: Running, Role: LEADER
I20260929 16:35:21.324477 13276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2e2e0dfdaf4756b0cc381c1a98a13b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46217 } }
I20260929 16:35:21.324782 13267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:21.325959 13279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a2e2e0dfdaf4756b0cc381c1a98a13b. Latest consensus state: current_term: 1 leader_uuid: "1a2e2e0dfdaf4756b0cc381c1a98a13b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2e2e0dfdaf4756b0cc381c1a98a13b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46217 } } }
I20260929 16:35:21.326082 13279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:21.326454 13286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:21.326417 13278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a2e2e0dfdaf4756b0cc381c1a98a13b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2e2e0dfdaf4756b0cc381c1a98a13b" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 46217 } } }
I20260929 16:35:21.326506 13278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:21.329448 13286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:21.332191 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:21.323661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45705 (local address 127.23.44.62:46217)
0929 16:35:21.323957 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:21.323961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:21.323976 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:35:21.324085 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:21.324088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:21.324126 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:21.324213 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:21.324973 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:21.325656 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:21.326835 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:21.327247 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:21.327322 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:21.328938 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:21.328942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:21.328945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:21.328967 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:21.330582 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:21.331048 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:21.331051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:21.331053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:21.331134 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:21.331398 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:21.331401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:21.331403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:21.331561 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:21.331681 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:21.331903 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 16:35:21.332018 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":218}
I20260929 16:35:21.334787 13286 catalog_manager.cc:1347] Generated new cluster ID: 33b153c9387c4f60b5054b211581d818
I20260929 16:35:21.334843 13286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:21.344835 13286 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:21.345922 13286 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:21.353091 13286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a2e2e0dfdaf4756b0cc381c1a98a13b: Generated new TSK 0
I20260929 16:35:21.353783 13286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:21.359575 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 13199
2026-09-29T16:35:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:21 dist-test-slave-mlmz krb5kdc[13298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:21 dist-test-slave-mlmz krb5kdc[13298](info): set up 2 sockets
Sep 29 16:35:21 dist-test-slave-mlmz krb5kdc[13298](info): commencing operation
krb5kdc: starting...
W20260929 16:35:22.389447 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:22 dist-test-slave-mlmz krb5kdc[13298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:35:22.505750 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:45211
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:45211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:22.611505 13311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:22.611747 13311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:22.611814 13311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:22.615705 13311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:22.615777 13311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:22.615818 13311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:22.615878 13311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:22.615917 13311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:22.620842 13311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:45211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:45211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:22.621768 13311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:22.622637 13311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:22.630119 13316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:22.630141 13319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:22.630141 13317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:22.630357 13311 server_base.cc:1034] running on GCE node
I20260929 16:35:22.630972 13311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:22.631930 13311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:22.633112 13311 hybrid_clock.cc:648] HybridClock initialized: now 1790699722633104 us; error 28 us; skew 500 ppm
Sep 29 16:35:22 dist-test-slave-mlmz krb5kdc[13298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699722, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:22.639484 13311 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:35:22.640271 13311 webserver.cc:460] Webserver started at http://127.23.44.62:38361/ using document root <none> and password file <none>
I20260929 16:35:22.640779 13311 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:22.640863 13311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:22.641064 13311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:22.642940 13311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "053f6ac0a0a049a097478022f5c5471a"
format_stamp: "Formatted at 2026-09-29 16:35:22 on dist-test-slave-mlmz"
server_key: "39db64855706ff5ba8c2e10083a14050"
server_key_iv: "0acadf74306a78450f367090bca3ba38"
server_key_version: "encryptionkey@0"
I20260929 16:35:22.643458 13311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "053f6ac0a0a049a097478022f5c5471a"
format_stamp: "Formatted at 2026-09-29 16:35:22 on dist-test-slave-mlmz"
server_key: "39db64855706ff5ba8c2e10083a14050"
server_key_iv: "0acadf74306a78450f367090bca3ba38"
server_key_version: "encryptionkey@0"
I20260929 16:35:22.646811 13311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:35:22.648972 13326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:22.650195 13311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:35:22.650339 13311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "053f6ac0a0a049a097478022f5c5471a"
format_stamp: "Formatted at 2026-09-29 16:35:22 on dist-test-slave-mlmz"
server_key: "39db64855706ff5ba8c2e10083a14050"
server_key_iv: "0acadf74306a78450f367090bca3ba38"
server_key_version: "encryptionkey@0"
I20260929 16:35:22.650441 13311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:22.663506 13311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:22.666468 13311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:22.666647 13311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:22.673408 13311 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:45211
I20260929 16:35:22.673434 13378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:45211 every 8 connection(s)
I20260929 16:35:22.674240 13311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:22.676633 13379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:22.680893 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 13311
I20260929 16:35:22.681039 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:22.681425 23728 external_mini_cluster.cc:1428] Setting key 13f14eaf7d2cd57182e8cb2aa98b6a7a
I20260929 16:35:22.681653 13379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a: Bootstrap starting.
I20260929 16:35:22.683933 13379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:22.684594 13379 log.cc:824] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:22.686432 13379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a: No bootstrap required, opened a new log
I20260929 16:35:22.689744 13379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053f6ac0a0a049a097478022f5c5471a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45211 } }
I20260929 16:35:22.689972 13379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:22.690059 13379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 053f6ac0a0a049a097478022f5c5471a, State: Initialized, Role: FOLLOWER
I20260929 16:35:22.690308 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:22.682665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56479 (local address 127.23.44.62:45211)
0929 16:35:22.683085 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:22.683099 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:22.683134 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:35:22.683773 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:22.683800 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:22.684176 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:22.684503 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:22.684928 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:22.686142 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:22.687122 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:22.687599 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:22.687783 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:22.688590 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:22.688595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:22.688607 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:22.688986 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:22.689038 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:22.689061 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:22.689172 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:22.689315 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 16:35:22.689514 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":128,"threads_started":1}
I20260929 16:35:22.690531 13379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053f6ac0a0a049a097478022f5c5471a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45211 } }
I20260929 16:35:22.690682 13379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:22.690748 13379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:22.690848 13379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:22.691715 13379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053f6ac0a0a049a097478022f5c5471a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45211 } }
I20260929 16:35:22.692078 13379 leader_election.cc:302] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 053f6ac0a0a049a097478022f5c5471a; no voters:
I20260929 16:35:22.692399 13379 leader_election.cc:288] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:22.692487 13384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:22.692745 13384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [term 1 LEADER]: Becoming Leader. State: Replica: 053f6ac0a0a049a097478022f5c5471a, State: Running, Role: LEADER
I20260929 16:35:22.692960 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:22.693091 13384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053f6ac0a0a049a097478022f5c5471a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45211 } }
I20260929 16:35:22.693601 13379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:22.694511 13386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 053f6ac0a0a049a097478022f5c5471a. Latest consensus state: current_term: 1 leader_uuid: "053f6ac0a0a049a097478022f5c5471a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053f6ac0a0a049a097478022f5c5471a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45211 } } }
I20260929 16:35:22.694650 13386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:22.694967 13385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "053f6ac0a0a049a097478022f5c5471a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053f6ac0a0a049a097478022f5c5471a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 45211 } } }
I20260929 16:35:22.695042 13385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a [sys.catalog]: This master's current role is: LEADER
W20260929 16:35:22.698351 13405 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:35:22.698441 13405 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:35:22.698544 13397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:22.699405 13397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:22.700189 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:22.695122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46829 (local address 127.23.44.62:45211)
0929 16:35:22.695256 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:22.695259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:22.696362 (+ 1103us) server_negotiation.cc:407] Connection header received
0929 16:35:22.696438 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:22.696440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:22.696487 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:22.696571 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:22.696785 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:22.697505 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:22.698612 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:22.698901 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:22.699009 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:22.699776 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:22.699779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:22.699782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:22.699797 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:22.699823 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:22.699836 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:22.699916 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:22.700022 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 16:35:22.700070 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:35:22.705351 13397 catalog_manager.cc:1347] Generated new cluster ID: f84f77c363164dde80b0f9bf7b50e18b
I20260929 16:35:22.705422 13397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:22.714649 13397 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:22.715451 13397 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:22.719858 13397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 053f6ac0a0a049a097478022f5c5471a: Generated new TSK 0
I20260929 16:35:22.720399 13397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:22.726848 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 13311
2026-09-29T16:35:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:22 dist-test-slave-mlmz krb5kdc[13410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:22 dist-test-slave-mlmz krb5kdc[13410](info): set up 2 sockets
Sep 29 16:35:22 dist-test-slave-mlmz krb5kdc[13410](info): commencing operation
krb5kdc: starting...
W20260929 16:35:23.757539 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:23 dist-test-slave-mlmz krb5kdc[13410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:35:23.874524 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:42483
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:42483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:23.982182 13423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:23.982424 13423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:23.982494 13423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:23.986478 13423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:23.986544 13423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:23.986564 13423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:23.986608 13423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:23.986639 13423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:23.991940 13423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:42483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:42483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:23.992945 13423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:23.993920 13423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:24.002305 13428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:24.002326 13429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:24.002321 13431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:24.002395 13423 server_base.cc:1034] running on GCE node
I20260929 16:35:24.003012 13423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:24.004124 13423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:24.005395 13423 hybrid_clock.cc:648] HybridClock initialized: now 1790699724005368 us; error 31 us; skew 500 ppm
Sep 29 16:35:24 dist-test-slave-mlmz krb5kdc[13410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699724, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:24.007994 13423 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:35:24.008819 13423 webserver.cc:460] Webserver started at http://127.23.44.62:41899/ using document root <none> and password file <none>
I20260929 16:35:24.009418 13423 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:24.009510 13423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:24.009737 13423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:24.011691 13423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "e41089b690ab4ae6abcf81c9185aeb6c"
format_stamp: "Formatted at 2026-09-29 16:35:24 on dist-test-slave-mlmz"
server_key: "c19696527919f980be8da05798d00265"
server_key_iv: "03c19af89f3df3ec98560cd479774e53"
server_key_version: "encryptionkey@0"
I20260929 16:35:24.012233 13423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "e41089b690ab4ae6abcf81c9185aeb6c"
format_stamp: "Formatted at 2026-09-29 16:35:24 on dist-test-slave-mlmz"
server_key: "c19696527919f980be8da05798d00265"
server_key_iv: "03c19af89f3df3ec98560cd479774e53"
server_key_version: "encryptionkey@0"
I20260929 16:35:24.015808 13423 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 16:35:24.018121 13438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:24.019359 13423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:35:24.019527 13423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "e41089b690ab4ae6abcf81c9185aeb6c"
format_stamp: "Formatted at 2026-09-29 16:35:24 on dist-test-slave-mlmz"
server_key: "c19696527919f980be8da05798d00265"
server_key_iv: "03c19af89f3df3ec98560cd479774e53"
server_key_version: "encryptionkey@0"
I20260929 16:35:24.019639 13423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:24.024036 13423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:24.026840 13423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:24.027029 13423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:24.033696 13423 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:42483
I20260929 16:35:24.033761 13490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:42483 every 8 connection(s)
I20260929 16:35:24.034598 13423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:24.037142 13491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:24.040108 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 13423
I20260929 16:35:24.040249 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:24.040534 23728 external_mini_cluster.cc:1428] Setting key ebbcbc785333d3aa94a78a7db2fa284f
I20260929 16:35:24.043005 13491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c: Bootstrap starting.
I20260929 16:35:24.045632 13491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:24.045719 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:24.041767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50651 (local address 127.23.44.62:42483)
0929 16:35:24.042167 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:24.042186 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:24.042227 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:35:24.043008 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:24.043029 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:24.043359 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:24.043693 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:24.044356 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:24.044362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:24.044376 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:24.044386 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:24.044440 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:24.044465 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:24.044597 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:24.044721 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 16:35:24.044857 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260929 16:35:24.046473 13491 log.cc:824] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:24.048450 13491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c: No bootstrap required, opened a new log
I20260929 16:35:24.048460 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:24.051967 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:24.050443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36525 (local address 127.23.44.62:42483)
0929 16:35:24.050544 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:24.050547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:24.050728 (+ 181us) server_negotiation.cc:407] Connection header received
0929 16:35:24.050846 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:24.050848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:24.050874 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:24.050941 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:24.051611 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:24.051614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:24.051616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:35:24.051625 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:24.051647 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:35:24.051656 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:24.051717 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:24.051835 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 16:35:24.051861 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 16:35:24.052871 13491 raft_consensus.cc:348] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41089b690ab4ae6abcf81c9185aeb6c" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42483 } }
I20260929 16:35:24.053237 13491 raft_consensus.cc:374] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:35:24.053324 13491 raft_consensus.cc:729] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41089b690ab4ae6abcf81c9185aeb6c, State: Initialized, Role: FOLLOWER
I20260929 16:35:24.053711 13491 consensus_queue.cc:260] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41089b690ab4ae6abcf81c9185aeb6c" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42483 } }
I20260929 16:35:24.053865 13491 raft_consensus.cc:388] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:35:24.053930 13491 raft_consensus.cc:482] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:35:24.054020 13491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:24.054865 13491 raft_consensus.cc:504] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41089b690ab4ae6abcf81c9185aeb6c" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42483 } }
I20260929 16:35:24.055197 13491 leader_election.cc:302] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e41089b690ab4ae6abcf81c9185aeb6c; no voters:
I20260929 16:35:24.055527 13491 leader_election.cc:288] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:35:24.055605 13501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:24.055859 13501 raft_consensus.cc:686] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [term 1 LEADER]: Becoming Leader. State: Replica: e41089b690ab4ae6abcf81c9185aeb6c, State: Running, Role: LEADER
I20260929 16:35:24.056190 13501 consensus_queue.cc:237] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41089b690ab4ae6abcf81c9185aeb6c" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42483 } }
I20260929 16:35:24.056684 13491 sys_catalog.cc:564] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:24.057811 13503 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e41089b690ab4ae6abcf81c9185aeb6c. Latest consensus state: current_term: 1 leader_uuid: "e41089b690ab4ae6abcf81c9185aeb6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41089b690ab4ae6abcf81c9185aeb6c" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42483 } } }
I20260929 16:35:24.057840 13502 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e41089b690ab4ae6abcf81c9185aeb6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41089b690ab4ae6abcf81c9185aeb6c" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 42483 } } }
I20260929 16:35:24.058002 13502 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:24.058274 13503 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:24.058434 13510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:24.061229 13510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:24.065979 13510 catalog_manager.cc:1347] Generated new cluster ID: 10b676677f0a4b2aaff9a077a82cc7bf
I20260929 16:35:24.066042 13510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:24.094192 13510 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:24.094492 13510 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:24.101027 13510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e41089b690ab4ae6abcf81c9185aeb6c: Generated new TSK 0
I20260929 16:35:24.101791 13510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:35:24.108449 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 13423
2026-09-29T16:35:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1382 ms)
[----------] 18 tests from AuthTokenIssuingTest (25072 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:24 dist-test-slave-mlmz krb5kdc[13522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:24 dist-test-slave-mlmz krb5kdc[13522](info): set up 2 sockets
Sep 29 16:35:24 dist-test-slave-mlmz krb5kdc[13522](info): commencing operation
krb5kdc: starting...
W20260929 16:35:25.149399 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:35:25.263427 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:43373
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:43373,127.23.44.61:44295,127.23.44.60:45287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:25.367934 13535 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:35:25.368172 13535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:25.368194 13535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:25.368229 13535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:25.372108 13535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:25.372155 13535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:25.372174 13535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:25.372190 13535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:25.372205 13535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:25.377197 13535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:43373,127.23.44.61:44295,127.23.44.60:45287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:43373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:25.378033 13535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:25.378875 13535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:25.386713 13540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:25.386771 13543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:25.386780 13541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:25.387086 13535 server_base.cc:1034] running on GCE node
I20260929 16:35:25.387591 13535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:25.388602 13535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:25.389786 13535 hybrid_clock.cc:648] HybridClock initialized: now 1790699725389771 us; error 29 us; skew 500 ppm
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:25.392277 13535 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:35:25.393004 13535 webserver.cc:460] Webserver started at http://127.23.44.62:38557/ using document root <none> and password file <none>
I20260929 16:35:25.393530 13535 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:25.393589 13535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:25.393733 13535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:25.395501 13535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "52ede79b900b45c7b39083c41e0b682a"
format_stamp: "Formatted at 2026-09-29 16:35:25 on dist-test-slave-mlmz"
server_key: "f2df5b572f931a992bf5089812000042"
server_key_iv: "466a8af2fa1977fef5a27d224f0dd070"
server_key_version: "encryptionkey@0"
I20260929 16:35:25.395967 13535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "52ede79b900b45c7b39083c41e0b682a"
format_stamp: "Formatted at 2026-09-29 16:35:25 on dist-test-slave-mlmz"
server_key: "f2df5b572f931a992bf5089812000042"
server_key_iv: "466a8af2fa1977fef5a27d224f0dd070"
server_key_version: "encryptionkey@0"
I20260929 16:35:25.399174 13535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:35:25.401405 13550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:25.402539 13535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:25.402633 13535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "52ede79b900b45c7b39083c41e0b682a"
format_stamp: "Formatted at 2026-09-29 16:35:25 on dist-test-slave-mlmz"
server_key: "f2df5b572f931a992bf5089812000042"
server_key_iv: "466a8af2fa1977fef5a27d224f0dd070"
server_key_version: "encryptionkey@0"
I20260929 16:35:25.402765 13535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:25.411707 13535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:25.414337 13535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:25.414530 13535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:25.420954 13535 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:43373
I20260929 16:35:25.421237 13602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:43373 every 8 connection(s)
I20260929 16:35:25.421923 13535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:25.424333 13603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:25.427578 13603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:35:25.428529 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 13535
I20260929 16:35:25.428663 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:25.428956 23728 external_mini_cluster.cc:1428] Setting key d8f5717d05b930b301df22b2382a2a68
WARNING: no policy specified for kudu/127.23.44.61@KRBTEST.COM; defaulting to no policy
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.61@KRBTEST.COM" created.
I20260929 16:35:25.441758 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.429722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:43373 (local address 127.0.0.1:52118)
0929 16:35:25.430157 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.430205 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:25.431017 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:25.431644 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:25.431649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:25.432058 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:25.432276 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.432288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.433514 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.433517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.434193 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.434202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.434807 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.434811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.434917 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.435515 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:25.435532 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:25.437045 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:25.438957 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.438967 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.438976 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.439187 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.439493 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.439498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.439499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.439605 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.440148 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:25.440154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:25.440285 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 16:35:25.440445 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 16:35:25.440826 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":102,"thread_start_us":136,"threads_started":1}
I20260929 16:35:25.441758 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.429722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.23.44.62:43373)
0929 16:35:25.430212 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.430225 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.430311 (+ 86us) server_negotiation.cc:407] Connection header received
0929 16:35:25.431303 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.431307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.431470 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.431645 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.432435 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.433400 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.434312 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.434664 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.434825 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.437189 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.437198 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.437214 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.437244 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.438816 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.439290 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.439296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.439297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.439401 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.439697 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.439700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.439701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.440032 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.440173 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.440515 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.440752 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"spinlock_wait_cycles":13184,"thread_start_us":138,"threads_started":1}
I20260929 16:35:25.444257 13603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:35:25.445137 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.444829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:44295 (local address 127.0.0.1:38698)
0929 16:35:25.444963 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.445044 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:44295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:35:25.445402 13552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.44.61:44295: connect: Connection refused (error 111)
I20260929 16:35:25.446245 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.446004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:44295 (local address 127.0.0.1:38710)
0929 16:35:25.446139 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.446194 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:44295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:35:25.446408 13603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.61:44295: Network error: Client connection negotiation failed: client connection to 127.23.44.61:44295: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.61.keytab.
Entry for principal kudu/127.23.44.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.61.keytab.
I20260929 16:35:25.453882 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.61:44295
--webserver_interface=127.23.44.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.61.keytab
--principal=kudu/127.23.44.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:43373,127.23.44.61:44295,127.23.44.60:45287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:35:25.495661 13603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } attempt: 1
I20260929 16:35:25.496585 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.496234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:44295 (local address 127.0.0.1:38726)
0929 16:35:25.496414 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.496496 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:44295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 16:35:25.497331 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.497093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:44295 (local address 127.0.0.1:38740)
0929 16:35:25.497197 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.497248 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:44295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 16:35:25.497467 13603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.61:44295: Network error: Client connection negotiation failed: client connection to 127.23.44.61:44295: connect: Connection refused (error 111)
W20260929 16:35:25.557449 13610 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:35:25.557680 13610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:25.557730 13610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:25.557806 13610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:25.561642 13610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:25.561684 13610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:25.561702 13610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:25.561745 13610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:25.561777 13610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:25.565629 13603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } attempt: 2
I20260929 16:35:25.566416 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.566116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:44295 (local address 127.0.0.1:38748)
0929 16:35:25.566270 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.566333 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:44295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:35:25.567132 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.566923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:44295 (local address 127.0.0.1:38758)
0929 16:35:25.567025 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.567074 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:44295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 16:35:25.567148 13610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:43373,127.23.44.61:44295,127.23.44.60:45287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.61.keytab
--principal=kudu/127.23.44.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.61:44295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.44.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
W20260929 16:35:25.567270 13603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.61:44295: Network error: Client connection negotiation failed: client connection to 127.23.44.61:44295: connect: Connection refused (error 111)
I20260929 16:35:25.568039 13610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:25.568928 13610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:25.576754 13615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:25.576748 13616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:25.576783 13618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:25.577018 13610 server_base.cc:1034] running on GCE node
I20260929 16:35:25.577598 13610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:25.578562 13610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:25.579736 13610 hybrid_clock.cc:648] HybridClock initialized: now 1790699725579712 us; error 45 us; skew 500 ppm
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:25.582383 13610 init.cc:377] Logged in from keytab as kudu/127.23.44.61@KRBTEST.COM (short username kudu)
I20260929 16:35:25.583205 13610 webserver.cc:460] Webserver started at http://127.23.44.61:38417/ using document root <none> and password file <none>
I20260929 16:35:25.583719 13610 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:25.583817 13610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:25.584033 13610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:25.585956 13610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/data/instance:
uuid: "1d4990d456c8459084819787bf6d80d9"
format_stamp: "Formatted at 2026-09-29 16:35:25 on dist-test-slave-mlmz"
server_key: "4aa3e0e945aac29f1d12135bc02d8f17"
server_key_iv: "f446480afdbc8af93ee1e2f5657fc415"
server_key_version: "encryptionkey@0"
I20260929 16:35:25.586494 13610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/wal/instance:
uuid: "1d4990d456c8459084819787bf6d80d9"
format_stamp: "Formatted at 2026-09-29 16:35:25 on dist-test-slave-mlmz"
server_key: "4aa3e0e945aac29f1d12135bc02d8f17"
server_key_iv: "f446480afdbc8af93ee1e2f5657fc415"
server_key_version: "encryptionkey@0"
I20260929 16:35:25.589984 13610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:35:25.592173 13625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:25.593386 13610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:35:25.593508 13610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/wal
uuid: "1d4990d456c8459084819787bf6d80d9"
format_stamp: "Formatted at 2026-09-29 16:35:25 on dist-test-slave-mlmz"
server_key: "4aa3e0e945aac29f1d12135bc02d8f17"
server_key_iv: "f446480afdbc8af93ee1e2f5657fc415"
server_key_version: "encryptionkey@0"
I20260929 16:35:25.593621 13610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:25.609125 13610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:25.612128 13610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:25.612319 13610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:25.618891 13610 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.61:44295
I20260929 16:35:25.619180 13677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.61:44295 every 8 connection(s)
I20260929 16:35:25.619788 13610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
I20260929 16:35:25.622361 13678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:25.625758 13678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:35:25.629354 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 13610
I20260929 16:35:25.629479 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-1/wal/instance
I20260929 16:35:25.629746 23728 external_mini_cluster.cc:1428] Setting key 6089cac36f80e8b537383971ea07a53d
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.61@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.60@KRBTEST.COM; defaulting to no policy
I20260929 16:35:25.638785 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.627473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52120 (local address 127.23.44.62:43373)
0929 16:35:25.627678 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.627682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.628350 (+ 668us) server_negotiation.cc:407] Connection header received
0929 16:35:25.629230 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.629234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.629319 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.629430 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.630563 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.631106 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.632172 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.632412 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.632527 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.635148 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.635151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.635154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.635183 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.636613 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.637166 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.637170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.637172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.637290 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.637671 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.637674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.637676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.637848 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.637956 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.638466 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.638632 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.60@KRBTEST.COM" created.
I20260929 16:35:25.639535 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.627789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:43373 (local address 127.0.0.1:52120)
0929 16:35:25.628194 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.628231 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:25.629030 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:25.629613 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:25.629621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:25.629991 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:25.630417 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.630429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.631230 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.631235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.632045 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.632053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.632560 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.632565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.632724 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.633343 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:25.633359 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:25.634986 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:25.636749 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.636754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.636767 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.637062 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.637409 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.637412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.637414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.637547 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.637954 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:25.637961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:25.638188 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 16:35:25.638366 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 16:35:25.638658 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
I20260929 16:35:25.641014 13678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.61@KRBTEST.COM for kudu/127.23.44.61@KRBTEST.COM
I20260929 16:35:25.650146 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.641492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:44295 (local address 127.0.0.1:38762)
0929 16:35:25.641595 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.641609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:25.641679 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:25.642388 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:25.642390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:25.642509 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:25.642586 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.642591 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.643538 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.643541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.644056 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.644063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.644629 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.644633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.644781 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.645216 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:25.645231 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:25.646645 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:25.648583 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.648586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.648588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.648794 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.649090 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.649093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.649095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.649173 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.649651 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:25.649654 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:25.649730 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 16:35:25.649864 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 16:35:25.649978 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":38}
I20260929 16:35:25.650193 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.641644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.23.44.61:44295)
0929 16:35:25.641982 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.641995 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.642019 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:35:25.642064 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.642072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.642250 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.642393 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.642694 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.643416 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.644167 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.644522 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.644633 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.646773 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.646782 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.646793 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.646820 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.648462 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.648922 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.648926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.648927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.648995 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.649286 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.649289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.649290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.649553 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.649684 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.649935 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.650050 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":227,"thread_start_us":130,"threads_started":1}
I20260929 16:35:25.651865 13678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:35:25.652454 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.652253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57568)
0929 16:35:25.652330 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.652395 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260929 16:35:25.652649 13629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.60.keytab.
Entry for principal kudu/127.23.44.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.60.keytab.
I20260929 16:35:25.653514 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.653185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57570)
0929 16:35:25.653390 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.653445 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260929 16:35:25.653684 13678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:45287: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
I20260929 16:35:25.654073 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.60:45287
--webserver_interface=127.23.44.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.60.keytab
--principal=kudu/127.23.44.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:43373,127.23.44.61:44295,127.23.44.60:45287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:35:25.658411 13603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } attempt: 3
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.61@KRBTEST.COM
I20260929 16:35:25.666596 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.658938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.23.44.61:44295)
0929 16:35:25.659039 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.659045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.659059 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:35:25.659202 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.659205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.659243 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.659328 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.659711 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.660202 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.661420 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.661665 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.661772 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.663613 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.663616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.663618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.663641 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.665000 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.665485 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.665489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.665490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.665583 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.665852 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.665856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.665858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.666016 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.666120 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.666365 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.666461 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 16:35:25.666606 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.658873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:44295 (local address 127.0.0.1:38774)
0929 16:35:25.658998 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.659012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:25.659105 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:25.659344 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:25.659346 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:25.659511 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:25.659599 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.659604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.660306 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.660310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.661249 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.661303 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.661774 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.661778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.661883 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.662334 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:25.662346 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:25.663486 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:25.665110 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.665115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.665117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.665389 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.665671 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.665676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.665678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.665763 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.666110 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:25.666114 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:25.666211 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 16:35:25.666316 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 16:35:25.666453 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 16:35:25.667634 13603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:35:25.668232 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.668021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57584)
0929 16:35:25.668127 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.668179 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 16:35:25.668895 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.668660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57590)
0929 16:35:25.668791 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.668840 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 16:35:25.669027 13603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:45287: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
I20260929 16:35:25.700178 13603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } attempt: 1
I20260929 16:35:25.700834 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.700592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57606)
0929 16:35:25.700713 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.700771 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:35:25.701572 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.701382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57622)
0929 16:35:25.701477 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.701521 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 16:35:25.701702 13603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:45287: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
I20260929 16:35:25.702875 13678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } attempt: 1
I20260929 16:35:25.703516 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.703223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57632)
0929 16:35:25.703382 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.703444 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 16:35:25.704188 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.703978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57636)
0929 16:35:25.704080 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.704130 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 16:35:25.704391 13678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:45287: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
W20260929 16:35:25.758600 13685 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:35:25.758828 13685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:25.758913 13685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:25.759003 13685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:25.762844 13685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:25.762893 13685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:25.762912 13685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:25.762928 13685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:25.762982 13685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:25.767858 13685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:43373,127.23.44.61:44295,127.23.44.60:45287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.60.keytab
--principal=kudu/127.23.44.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.60:45287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.44.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:25.768705 13685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:25.769595 13685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:35:25.772569 13678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } attempt: 2
I20260929 16:35:25.773499 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.773077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57652)
0929 16:35:25.773314 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.773378 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260929 16:35:25.774196 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.773994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57664)
0929 16:35:25.774098 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.774149 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:35:25.774318 13678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:45287: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
I20260929 16:35:25.776922 13603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } attempt: 2
W20260929 16:35:25.777577 13693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:25.777678 13685 server_base.cc:1034] running on GCE node
W20260929 16:35:25.777599 13691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:25.777874 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.777590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57666)
0929 16:35:25.777737 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.777796 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:35:25.777769 13690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:25.778484 13685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:25.778744 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.778516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57676)
0929 16:35:25.778636 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.778688 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:35:25.778875 13603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:45287: Network error: Client connection negotiation failed: client connection to 127.23.44.60:45287: connect: Connection refused (error 111)
I20260929 16:35:25.779569 13685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:25.780764 13685 hybrid_clock.cc:648] HybridClock initialized: now 1790699725780740 us; error 44 us; skew 500 ppm
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:25.783399 13685 init.cc:377] Logged in from keytab as kudu/127.23.44.60@KRBTEST.COM (short username kudu)
I20260929 16:35:25.784227 13685 webserver.cc:460] Webserver started at http://127.23.44.60:38283/ using document root <none> and password file <none>
I20260929 16:35:25.784683 13685 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:25.784736 13685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:25.784899 13685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:25.786758 13685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/data/instance:
uuid: "5f2fe099acf44c149d1314b6ff256303"
format_stamp: "Formatted at 2026-09-29 16:35:25 on dist-test-slave-mlmz"
server_key: "a46a169cb48d8abf905840cabc1d6570"
server_key_iv: "7dc537ba1ea632ab1120016563a6a8ec"
server_key_version: "encryptionkey@0"
I20260929 16:35:25.787225 13685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/wal/instance:
uuid: "5f2fe099acf44c149d1314b6ff256303"
format_stamp: "Formatted at 2026-09-29 16:35:25 on dist-test-slave-mlmz"
server_key: "a46a169cb48d8abf905840cabc1d6570"
server_key_iv: "7dc537ba1ea632ab1120016563a6a8ec"
server_key_version: "encryptionkey@0"
I20260929 16:35:25.790508 13685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:35:25.792647 13700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:25.793817 13685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:25.793916 13685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/wal
uuid: "5f2fe099acf44c149d1314b6ff256303"
format_stamp: "Formatted at 2026-09-29 16:35:25 on dist-test-slave-mlmz"
server_key: "a46a169cb48d8abf905840cabc1d6570"
server_key_iv: "7dc537ba1ea632ab1120016563a6a8ec"
server_key_version: "encryptionkey@0"
I20260929 16:35:25.793993 13685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:25.814953 13685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:25.817413 13685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:25.817543 13685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:25.823975 13685 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.60:45287
I20260929 16:35:25.824244 13752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.60:45287 every 8 connection(s)
I20260929 16:35:25.824842 13685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
I20260929 16:35:25.827369 13753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:25.829209 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 13685
I20260929 16:35:25.829346 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699450427356-23728-0/minicluster-data/master-2/wal/instance
I20260929 16:35:25.829619 23728 external_mini_cluster.cc:1428] Setting key 8e403cb69ea7a095ba726ae096374f5a
I20260929 16:35:25.831351 13753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.60@KRBTEST.COM
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.60@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:35:25.842917 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.833386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.23.44.62:43373)
0929 16:35:25.833511 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.833516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.833773 (+ 257us) server_negotiation.cc:407] Connection header received
0929 16:35:25.833897 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.833902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.833957 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.834028 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.834832 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.835525 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.836661 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.836993 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.837097 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.839745 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.839748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.839751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.839783 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.841089 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.841599 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.841601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.841602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.841674 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.842009 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.842012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.842013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.842184 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.842258 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.842644 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.842754 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:35:25.843592 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.833215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:43373 (local address 127.0.0.1:52132)
0929 16:35:25.833647 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.833692 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:25.833802 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:25.834105 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:25.834110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:25.834520 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:25.834692 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.834703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.835640 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.835642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.836501 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.836511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.837139 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.837143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.837291 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.837808 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:25.837823 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:25.839605 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:25.841207 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.841211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.841217 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.841502 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.841790 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.841792 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.841793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.841891 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.842282 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:25.842287 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:25.842406 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 16:35:25.842560 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:35:25.842839 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":186,"threads_started":1}
I20260929 16:35:25.843597 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.830901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.23.44.60:45287)
0929 16:35:25.831282 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.831296 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.831328 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:35:25.832028 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.832043 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.832394 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.832704 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.833561 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.834952 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.835966 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.836568 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.836710 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.839233 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.839238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.839246 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.839266 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.841054 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.841512 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.841518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.841522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.841639 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.841961 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.841967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.841969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.842322 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.842527 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.842685 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.842983 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":262,"thread_start_us":127,"threads_started":1}
I20260929 16:35:25.844718 13753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.60@KRBTEST.COM for kudu/127.23.44.61@KRBTEST.COM
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:35:25.852561 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.845155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:44295 (local address 127.0.0.1:38776)
0929 16:35:25.845318 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.845331 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:25.845393 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:25.845641 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:25.845644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:25.845801 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:25.845890 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.845896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.846604 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.846610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.847208 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.847216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.847647 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.847651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.847729 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.848155 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:25.848168 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:25.849327 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:25.851010 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.851015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.851018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.851210 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.851506 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.851509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.851511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.851603 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.852053 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:25.852057 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:25.852158 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 16:35:25.852280 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 16:35:25.852388 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 16:35:25.852653 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.845250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.23.44.61:44295)
0929 16:35:25.845395 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.845399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.845414 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:35:25.845463 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.845466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.845516 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.845605 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.846026 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.846499 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.847326 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.847542 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.847646 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.849470 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.849473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.849476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.849504 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.850899 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.851309 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.851312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.851314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.851414 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.851700 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.851703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.851704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.851950 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.852052 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.852369 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.852501 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:35:25.853231 13753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:35:25.854844 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.845923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.23.44.62:43373)
0929 16:35:25.846682 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.846685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.846696 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:35:25.846726 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.846728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.846763 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.846839 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.847399 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.847835 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.848582 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.848782 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.848888 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.851499 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.851503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.851505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.851529 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.853193 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.853651 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.853654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.853656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.853744 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.854037 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.854041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.854044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.854247 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.854413 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.854603 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.854706 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690}
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.60@KRBTEST.COM for kudu/127.23.44.60@KRBTEST.COM
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.61@KRBTEST.COM
I20260929 16:35:25.861709 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.853710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57684)
0929 16:35:25.853843 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.853856 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:25.853951 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:25.854198 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:25.854200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:25.854344 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:25.854423 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.854430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.855263 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.855266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.856014 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.856026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.856636 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.856638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.856694 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.857067 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:25.857080 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:25.858226 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:25.860123 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.860125 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.860127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.860346 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.860655 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.860659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.860660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.860747 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.861166 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:25.861169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:25.861252 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 16:35:25.861411 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 16:35:25.861550 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 16:35:25.861734 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.853898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.23.44.60:45287)
0929 16:35:25.854010 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.854016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.854029 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:35:25.854058 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.854061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.854102 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.854177 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.854560 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.855181 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.856124 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.856515 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.856622 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.858347 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.858352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.858356 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.858395 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.860007 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.860477 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.860480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.860481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.860553 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.860834 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.860836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.860837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.861060 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.861165 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.861466 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.861602 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":40}
I20260929 16:35:25.864445 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.855879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.23.44.61:44295)
0929 16:35:25.855967 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.855971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.855984 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:35:25.856054 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.856059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.856108 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.856179 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.856858 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.857613 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.858431 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.858624 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.858695 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.861325 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.861328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.861331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.861353 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.862937 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.863338 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.863346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.863347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.863416 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.863680 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.863685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.863687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.863863 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.864091 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.864197 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.864304 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 16:35:25.865031 13753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303: Bootstrap starting.
I20260929 16:35:25.865479 13678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } attempt: 3
I20260929 16:35:25.867515 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:25.867707 13753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:25.868432 13753 log.cc:824] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:25.868847 13577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "52ede79b900b45c7b39083c41e0b682a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52134
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.61@KRBTEST.COM for kudu/127.23.44.60@KRBTEST.COM
I20260929 16:35:25.870594 13753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303: No bootstrap required, opened a new log
I20260929 16:35:25.874220 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.865817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57700)
0929 16:35:25.865920 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.865934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:25.866008 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:25.866390 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:25.866393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:25.866544 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:25.866651 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.866658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.867520 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.867524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.868284 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.868294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.868899 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.868902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.868996 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.869445 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:25.869458 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:25.870816 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:25.872665 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.872668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.872671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.872923 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.873199 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.873202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.873206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.873326 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.873754 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:25.873756 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:25.873842 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 16:35:25.873947 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 16:35:25.874063 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 16:35:25.874488 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.866106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.23.44.60:45287)
0929 16:35:25.866213 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.866216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.866225 (+ 9us) server_negotiation.cc:407] Connection header received
0929 16:35:25.866251 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.866253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.866281 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.866368 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.866765 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.867413 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.868406 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.868796 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.868896 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.870955 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.870959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.870962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.870985 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.872548 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.873021 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.873024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.873026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.873106 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.873417 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.873422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.873425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.873653 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.873774 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.874013 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.874122 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 16:35:25.874836 13753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } }
I20260929 16:35:25.875100 13753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2fe099acf44c149d1314b6ff256303, State: Initialized, Role: FOLLOWER
I20260929 16:35:25.875422 13753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } }
I20260929 16:35:25.875933 13759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } } }
I20260929 16:35:25.876127 13759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:25.876708 13753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:25.876951 13678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9: Bootstrap starting.
I20260929 16:35:25.878024 13603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } attempt: 3
I20260929 16:35:25.879652 13678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:25.880466 13678 log.cc:824] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9: Log is configured to *not* fsync() on all Append() calls
W20260929 16:35:25.881529 13770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:35:25.881587 13770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:35:25.882654 13678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9: No bootstrap required, opened a new log
Sep 29 16:35:25 dist-test-slave-mlmz krb5kdc[13522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.60@KRBTEST.COM
I20260929 16:35:25.881994 13685 master_runner.cc:186] Error getting master registration for 127.23.44.62:43373: OK, instance_id { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" instance_seqno: 1790699725419139 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:35:25.885401 13685 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:35:25.887013 13678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } }
I20260929 16:35:25.887221 13678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4990d456c8459084819787bf6d80d9, State: Initialized, Role: FOLLOWER
I20260929 16:35:25.887364 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.878409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:45287 (local address 127.0.0.1:57714)
0929 16:35:25.878595 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:25.878610 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:25.878705 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:25.879124 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:25.879126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:25.879303 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:25.879397 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.879403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.880505 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.880508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.881318 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:25.881329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.881905 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.881908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:25.882004 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.882484 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:25.882498 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:25.883899 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:25.886012 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.886015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.886016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.886202 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.886480 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:25.886482 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:25.886483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:25.886554 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:25.886918 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:25.886920 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:25.886998 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 16:35:25.887075 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 16:35:25.887171 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260929 16:35:25.887392 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.878446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.23.44.60:45287)
0929 16:35:25.878536 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.878540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.878656 (+ 116us) server_negotiation.cc:407] Connection header received
0929 16:35:25.878823 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.878827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.878871 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.878942 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.879512 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.880164 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.881455 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.881787 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.881881 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.884075 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.884079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.884081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.884104 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.885906 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.886291 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.886295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.886297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.886385 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.886634 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.886638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.886639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.886827 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.886924 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.887144 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.887252 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 16:35:25.887535 13678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } }
I20260929 16:35:25.887987 13773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } } }
I20260929 16:35:25.888121 13773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:25.888700 13678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:25.889884 13603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a: Bootstrap starting.
I20260929 16:35:25.892434 13603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:25.893244 13603 log.cc:824] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a: Log is configured to *not* fsync() on all Append() calls
W20260929 16:35:25.893826 13784 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:35:25.893944 13784 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:35:25.891906 13610 master_runner.cc:186] Error getting master registration for 127.23.44.62:43373: OK, instance_id { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" instance_seqno: 1790699725419139 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:35:25.895180 13603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a: No bootstrap required, opened a new log
I20260929 16:35:25.895229 13610 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:35:25.897993 13603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } }
I20260929 16:35:25.898181 13603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ede79b900b45c7b39083c41e0b682a, State: Initialized, Role: FOLLOWER
I20260929 16:35:25.898408 13603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } }
I20260929 16:35:25.898774 13787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } } }
I20260929 16:35:25.898893 13787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:25.899436 13603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:25.899546 13577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:35:25.899658 13577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:25.900853 13577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } }
I20260929 16:35:25.902225 13652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52ede79b900b45c7b39083c41e0b682a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1d4990d456c8459084819787bf6d80d9"
I20260929 16:35:25.902419 13652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:25.903669 13652 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ede79b900b45c7b39083c41e0b682a in term 1.
I20260929 16:35:25.904161 13552 leader_election.cc:302] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4990d456c8459084819787bf6d80d9, 52ede79b900b45c7b39083c41e0b682a; no voters:
I20260929 16:35:25.904422 13787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [term 1 FOLLOWER]: Leader election won for term 1
W20260929 16:35:25.904438 13798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:35:25.904521 13798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:35:25.904676 13787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [term 1 LEADER]: Becoming Leader. State: Replica: 52ede79b900b45c7b39083c41e0b682a, State: Running, Role: LEADER
I20260929 16:35:25.904987 13787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } }
I20260929 16:35:25.905884 13727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52ede79b900b45c7b39083c41e0b682a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5f2fe099acf44c149d1314b6ff256303"
I20260929 16:35:25.906106 13727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:25.907147 13800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52ede79b900b45c7b39083c41e0b682a. Latest consensus state: current_term: 1 leader_uuid: "52ede79b900b45c7b39083c41e0b682a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } } }
I20260929 16:35:25.907261 13800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:25.907368 13727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ede79b900b45c7b39083c41e0b682a in term 1.
I20260929 16:35:25.907658 13802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:25.907747 13577 leader_election.cc:288] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [CANDIDATE]: Term 1 election: Requested vote from peers 1d4990d456c8459084819787bf6d80d9 (127.23.44.61:44295), 5f2fe099acf44c149d1314b6ff256303 (127.23.44.60:45287)
I20260929 16:35:25.908655 13802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:25.909410 13535 master_runner.cc:263] No leader master found from master 52ede79b900b45c7b39083c41e0b682a
I20260929 16:35:25.909484 13535 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 16:35:25.915817 13556 tablet.cc:2369] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:35:25.919812 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.910015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.23.44.60:45287)
0929 16:35:25.910130 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.910134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.910534 (+ 400us) server_negotiation.cc:407] Connection header received
0929 16:35:25.910720 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.910723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.910764 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.910856 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.911535 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.912160 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.913304 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.913653 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.913739 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.915518 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.915522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.915525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.915550 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.917825 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.918330 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.918334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.918337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.918432 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.918794 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.918798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.918801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.919035 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.919140 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.919512 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.919622 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 16:35:25.921669 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.911950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.23.44.61:44295)
0929 16:35:25.912059 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.912064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.912650 (+ 586us) server_negotiation.cc:407] Connection header received
0929 16:35:25.912745 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.912748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.912789 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.912862 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.913774 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.914505 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.915655 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.916013 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.916095 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.917989 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.917995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.918000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.918029 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.919778 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.920321 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.920327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.920331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.920416 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.920720 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.920723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.920725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.920928 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.921019 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.921416 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.921525 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 16:35:25.924926 13727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [term 1 FOLLOWER]: Refusing update from remote peer 52ede79b900b45c7b39083c41e0b682a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:25.924962 13652 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [term 1 FOLLOWER]: Refusing update from remote peer 52ede79b900b45c7b39083c41e0b682a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:25.925561 13787 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:25.925942 13787 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:25.928877 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:25.912334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.23.44.62:43373)
0929 16:35:25.912451 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:25.912456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:25.920557 (+ 8101us) server_negotiation.cc:407] Connection header received
0929 16:35:25.920722 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:25.920725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:25.920783 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:25.920836 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:25.921579 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.922288 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.923289 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:25.923502 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:25.923605 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:25.925386 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:25.925390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:25.925393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:25.925421 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:25.927102 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.927582 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.927586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.927587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.927668 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:25.927965 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:25.927971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:25.927973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:25.928171 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:25.928289 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:25.928634 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 16:35:25.928744 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 16:35:25.929587 13759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52ede79b900b45c7b39083c41e0b682a. Latest consensus state: current_term: 1 leader_uuid: "52ede79b900b45c7b39083c41e0b682a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } } }
I20260929 16:35:25.929702 13759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:25.930335 13773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52ede79b900b45c7b39083c41e0b682a. Latest consensus state: current_term: 1 leader_uuid: "52ede79b900b45c7b39083c41e0b682a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } } }
I20260929 16:35:25.930451 13773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:25.935397 13787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52ede79b900b45c7b39083c41e0b682a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } } }
I20260929 16:35:25.935518 13787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:25.936440 13759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52ede79b900b45c7b39083c41e0b682a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } } }
I20260929 16:35:25.936537 13759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:25.936946 13802 catalog_manager.cc:1347] Generated new cluster ID: 892a67f6c9a7409f8780333dc4fa8a35
I20260929 16:35:25.937004 13802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:25.937054 13808 mvcc.cc:205] Tried to move back new op lower bound from 7334706077382500352 to 7334706077310636032. Current Snapshot: MvccSnapshot[applied={T|T < 7334706077382500352 or (T in {7334706077382500352})}]
I20260929 16:35:25.937278 13812 mvcc.cc:205] Tried to move back new op lower bound from 7334706077382500352 to 7334706077310636032. Current Snapshot: MvccSnapshot[applied={T|T < 7334706077382500352}]
I20260929 16:35:25.950021 13787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52ede79b900b45c7b39083c41e0b682a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } } }
I20260929 16:35:25.950126 13787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:25.951086 13773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52ede79b900b45c7b39083c41e0b682a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ede79b900b45c7b39083c41e0b682a" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 43373 } } peers { permanent_uuid: "1d4990d456c8459084819787bf6d80d9" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 44295 } } peers { permanent_uuid: "5f2fe099acf44c149d1314b6ff256303" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 45287 } } }
I20260929 16:35:25.951188 13773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:25.956431 13802 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:25.957558 13802 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:25.976506 13802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52ede79b900b45c7b39083c41e0b682a: Generated new TSK 0
I20260929 16:35:25.977342 13802 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:35:26.010103 13756 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.007305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.23.44.60:45287)
0929 16:35:26.007419 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.007423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.009100 (+ 1677us) server_negotiation.cc:407] Connection header received
0929 16:35:26.009309 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.009313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.009358 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.009409 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.009979 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57744: BlockingRecv error: recv got EOF from 127.0.0.1:57744 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 16:35:26.011503 13683 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.007993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.23.44.61:44295)
0929 16:35:26.008096 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.008100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.010689 (+ 2589us) server_negotiation.cc:407] Connection header received
0929 16:35:26.010842 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.010845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.010887 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.010934 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.011380 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38800: BlockingRecv error: recv got EOF from 127.0.0.1:38800 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 16:35:26.013010 13606 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.008567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.23.44.62:43373)
0929 16:35:26.008669 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.008673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.012187 (+ 3514us) server_negotiation.cc:407] Connection header received
0929 16:35:26.012350 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.012353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.012396 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.012459 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.012903 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52154: BlockingRecv error: recv got EOF from 127.0.0.1:52154 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 16:35:26.017189 13683 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.015804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.23.44.61:44295)
0929 16:35:26.015895 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.015899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.016312 (+ 413us) server_negotiation.cc:407] Connection header received
0929 16:35:26.016524 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.016527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.016607 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.016656 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.017105 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38816: BlockingRecv error: recv got EOF from 127.0.0.1:38816 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 16:35:26.021982 13683 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.020648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.23.44.61:44295)
0929 16:35:26.020737 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.020741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.021179 (+ 438us) server_negotiation.cc:407] Connection header received
0929 16:35:26.021374 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.021377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.021429 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.021477 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.021901 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38826: BlockingRecv error: recv got EOF from 127.0.0.1:38826 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 16:35:26.027343 13683 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.026022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.23.44.61:44295)
0929 16:35:26.026135 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.026141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.026588 (+ 447us) server_negotiation.cc:407] Connection header received
0929 16:35:26.026755 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.026758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.026799 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.026844 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.027261 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38830: BlockingRecv error: recv got EOF from 127.0.0.1:38830 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 16:35:26.034709 13683 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.033308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.23.44.61:44295)
0929 16:35:26.033435 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.033439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.033930 (+ 491us) server_negotiation.cc:407] Connection header received
0929 16:35:26.034094 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.034097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.034138 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.034183 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.034626 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38834: BlockingRecv error: recv got EOF from 127.0.0.1:38834 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 16:35:26.046204 13683 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.044742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.23.44.61:44295)
0929 16:35:26.044833 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.044837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.045371 (+ 534us) server_negotiation.cc:407] Connection header received
0929 16:35:26.045565 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.045568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.045617 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.045665 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.046123 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38840: BlockingRecv error: recv got EOF from 127.0.0.1:38840 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260929 16:35:26.066179 13683 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.064562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.23.44.61:44295)
0929 16:35:26.064667 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.064671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.065191 (+ 520us) server_negotiation.cc:407] Connection header received
0929 16:35:26.065441 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.065445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.065495 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.065548 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.066080 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38850: BlockingRecv error: recv got EOF from 127.0.0.1:38850 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 16:35:26.101892 13683 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.100574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.23.44.61:44295)
0929 16:35:26.100706 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.100711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.101031 (+ 320us) server_negotiation.cc:407] Connection header received
0929 16:35:26.101191 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.101194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.101250 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.101344 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.101775 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38862: BlockingRecv error: recv got EOF from 127.0.0.1:38862 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 16:35:26.169049 13683 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.167897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.23.44.61:44295)
0929 16:35:26.168066 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.168071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.168203 (+ 132us) server_negotiation.cc:407] Connection header received
0929 16:35:26.168368 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.168372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.168425 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.168515 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.168952 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38868: BlockingRecv error: recv got EOF from 127.0.0.1:38868 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 16:35:26.300377 13683 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.299124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.23.44.61:44295)
0929 16:35:26.299306 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.299311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.299474 (+ 163us) server_negotiation.cc:407] Connection header received
0929 16:35:26.299649 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.299658 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.299706 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.299826 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.300247 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38882: BlockingRecv error: recv got EOF from 127.0.0.1:38882 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 16:35:26.559544 13683 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:26.558455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.23.44.61:44295)
0929 16:35:26.558594 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:26.558599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:26.558639 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:35:26.558820 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:26.558823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:26.558876 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:26.558956 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:26.559426 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38898: BlockingRecv error: recv got EOF from 127.0.0.1:38898 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:35:26.882814 13770 catalog_manager.cc:1259] Loaded cluster ID: 892a67f6c9a7409f8780333dc4fa8a35
I20260929 16:35:26.882903 13770 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303: loading cluster ID for follower catalog manager: success
I20260929 16:35:26.884508 13770 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303: acquiring CA information for follower catalog manager: success
I20260929 16:35:26.885593 13770 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f2fe099acf44c149d1314b6ff256303: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:35:26.895089 13784 catalog_manager.cc:1259] Loaded cluster ID: 892a67f6c9a7409f8780333dc4fa8a35
I20260929 16:35:26.895150 13784 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9: loading cluster ID for follower catalog manager: success
I20260929 16:35:26.896735 13784 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9: acquiring CA information for follower catalog manager: success
I20260929 16:35:26.897926 13784 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d4990d456c8459084819787bf6d80d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:35:27.077487 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:27.073730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.23.44.61:44295)
0929 16:35:27.073961 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:27.073965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:27.073979 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:35:27.074103 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:27.074105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:27.074223 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:27.074320 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:27.074527 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.075038 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.075864 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.076077 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.076152 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:27.076492 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:27.076988 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:27.077120 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:27.077183 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 16:35:27.077283 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260929 16:35:27.083923 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:27.080410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.23.44.60:45287)
0929 16:35:27.080644 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:27.080649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:27.080663 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:35:27.080792 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:27.080797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:27.080901 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:27.081019 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:27.081244 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.081757 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.082570 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.082792 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.082901 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:27.083197 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:27.083522 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:27.083647 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:27.083710 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 16:35:27.083777 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":60,"threads_started":1}
I20260929 16:35:27.090504 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:27.086578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.23.44.61:44295)
0929 16:35:27.086911 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:27.086916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:27.086930 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:35:27.087062 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:27.087065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:27.087116 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:27.087198 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:27.087401 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.088097 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.089239 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.089604 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.089706 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:27.090066 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:27.090137 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:27.090184 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:27.090344 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 16:35:27.090389 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260929 16:35:27.091060 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:27.086578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.23.44.60:45287)
0929 16:35:27.086675 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:27.086679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:27.087754 (+ 1075us) server_negotiation.cc:407] Connection header received
0929 16:35:27.087832 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:27.087835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:27.087877 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:27.087950 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:27.088198 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.088779 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.089790 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.090109 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.090218 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:27.090454 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:27.090519 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:27.090564 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:27.090905 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 16:35:27.090955 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 16:35:27.091325 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:27.086682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.23.44.62:43373)
0929 16:35:27.087084 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:27.087088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:27.087100 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:35:27.087177 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:27.087183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:27.087317 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:27.087441 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:27.087631 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.088427 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.089545 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:27.089932 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:27.090071 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:27.090378 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:27.090650 (+ 272us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:27.090730 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:27.091111 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 16:35:27.091170 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":102,"threads_started":1}
I20260929 16:35:27.094238 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 13535
I20260929 16:35:27.100322 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 13610
I20260929 16:35:27.105985 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 13685
2026-09-29T16:35:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2998 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:35:27 dist-test-slave-mlmz krb5kdc[13901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:35:27 dist-test-slave-mlmz krb5kdc[13901](info): set up 2 sockets
Sep 29 16:35:27 dist-test-slave-mlmz krb5kdc[13901](info): commencing operation
krb5kdc: starting...
W20260929 16:35:28.137567 23728 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/test-user.keytab.
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.44.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
Entry for principal kudu/127.23.44.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab.
I20260929 16:35:28.252249 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:40083
--webserver_interface=127.23.44.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:40083,127.23.44.61:38969,127.23.44.60:39785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:35:28.357141 13914 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:35:28.357455 13914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:28.357482 13914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:28.357517 13914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:28.361317 13914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:28.361359 13914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:28.361377 13914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:28.361393 13914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:28.361413 13914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:28.366294 13914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:40083,127.23.44.61:38969,127.23.44.60:39785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.62.keytab
--principal=kudu/127.23.44.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.62:40083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.44.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:28.367147 13914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:28.368014 13914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:28.376394 13919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:28.376399 13922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:28.376413 13920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:28.376761 13914 server_base.cc:1034] running on GCE node
I20260929 16:35:28.377334 13914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:28.378357 13914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:28.379514 13914 hybrid_clock.cc:648] HybridClock initialized: now 1790699728379487 us; error 38 us; skew 500 ppm
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:28.388411 13914 init.cc:377] Logged in from keytab as kudu/127.23.44.62@KRBTEST.COM (short username kudu)
I20260929 16:35:28.389525 13914 webserver.cc:460] Webserver started at http://127.23.44.62:40385/ using document root <none> and password file <none>
I20260929 16:35:28.390095 13914 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:28.390157 13914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:28.390313 13914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:28.392182 13914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/data/instance:
uuid: "042a8d36c5654fbea6b6c543838483fe"
format_stamp: "Formatted at 2026-09-29 16:35:28 on dist-test-slave-mlmz"
server_key: "29717d7e8e8955c9804384caaac49111"
server_key_iv: "0f39c60b4d6f4bbd8bdefde695c9d98f"
server_key_version: "encryptionkey@0"
I20260929 16:35:28.392670 13914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/wal/instance:
uuid: "042a8d36c5654fbea6b6c543838483fe"
format_stamp: "Formatted at 2026-09-29 16:35:28 on dist-test-slave-mlmz"
server_key: "29717d7e8e8955c9804384caaac49111"
server_key_iv: "0f39c60b4d6f4bbd8bdefde695c9d98f"
server_key_version: "encryptionkey@0"
I20260929 16:35:28.396026 13914 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:35:28.398320 13929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:28.399441 13914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:35:28.399534 13914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/wal
uuid: "042a8d36c5654fbea6b6c543838483fe"
format_stamp: "Formatted at 2026-09-29 16:35:28 on dist-test-slave-mlmz"
server_key: "29717d7e8e8955c9804384caaac49111"
server_key_iv: "0f39c60b4d6f4bbd8bdefde695c9d98f"
server_key_version: "encryptionkey@0"
I20260929 16:35:28.399610 13914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:28.435914 13914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:28.439016 13914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:28.439165 13914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:28.445832 13914 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.62:40083
I20260929 16:35:28.446118 13981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.62:40083 every 8 connection(s)
I20260929 16:35:28.446825 13914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/data/info.pb
I20260929 16:35:28.447599 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 13914
I20260929 16:35:28.447690 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-0/wal/instance
I20260929 16:35:28.447973 23728 external_mini_cluster.cc:1428] Setting key 035b5754a4a37fe3aa69aee080eebb3b
I20260929 16:35:28.449589 13982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:28.454751 13982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.44.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.61@KRBTEST.COM" created.
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:35:28.469732 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.457342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40083 (local address 127.0.0.1:42404)
0929 16:35:28.457817 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.457855 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:28.458618 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:28.459437 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:28.459442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:28.459728 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:28.460033 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.460046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.461292 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.461296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.461994 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.462002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.462594 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.462598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.462702 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.463273 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:28.463289 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:28.464702 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:28.466996 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.467003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.467010 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.467223 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.467596 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.467599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.467600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.467723 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.468293 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:28.468297 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:28.468450 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 16:35:28.468627 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 16:35:28.468899 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":21,"thread_start_us":113,"threads_started":1}
I20260929 16:35:28.469738 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.457342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.23.44.62:40083)
0929 16:35:28.457775 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.457792 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.457962 (+ 170us) server_negotiation.cc:407] Connection header received
0929 16:35:28.458979 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.458983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.459229 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.459424 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.460176 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.461144 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.462103 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.462447 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.462613 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.464856 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.464865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.464879 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.464911 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.466870 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.467337 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.467343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.467345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.467478 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.467831 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.467837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.467839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.468182 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.468287 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.468725 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.469036 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":188,"server-negotiator.queue_time_us":316,"spinlock_wait_cycles":37120,"thread_start_us":147,"threads_started":1}
I20260929 16:35:28.472007 13982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:35:28.472695 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.472471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:38969 (local address 127.0.0.1:54766)
0929 16:35:28.472583 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.472646 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:38969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:35:28.472925 13930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.44.61:38969: connect: Connection refused (error 111)
I20260929 16:35:28.473781 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.473557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:38969 (local address 127.0.0.1:54780)
0929 16:35:28.473649 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.473712 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:38969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 16:35:28.474015 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.61:38969: Network error: Client connection negotiation failed: client connection to 127.23.44.61:38969: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.61.keytab.
Entry for principal kudu/127.23.44.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.61.keytab.
I20260929 16:35:28.475471 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.61:38969
--webserver_interface=127.23.44.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.61.keytab
--principal=kudu/127.23.44.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:40083,127.23.44.61:38969,127.23.44.60:39785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:35:28.523280 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } attempt: 1
I20260929 16:35:28.524240 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.523884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:38969 (local address 127.0.0.1:54790)
0929 16:35:28.524080 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.524152 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:38969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 16:35:28.525092 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.524877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:38969 (local address 127.0.0.1:54804)
0929 16:35:28.524982 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.525032 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:38969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 16:35:28.525239 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.61:38969: Network error: Client connection negotiation failed: client connection to 127.23.44.61:38969: connect: Connection refused (error 111)
W20260929 16:35:28.586602 13989 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:35:28.586851 13989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:28.586874 13989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:28.586910 13989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:28.590821 13989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:28.590873 13989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:28.590893 13989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:28.590909 13989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:28.590924 13989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:28.593421 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } attempt: 2
I20260929 16:35:28.594209 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.593897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:38969 (local address 127.0.0.1:54814)
0929 16:35:28.594061 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.594125 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:38969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:35:28.594883 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.594672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:38969 (local address 127.0.0.1:54828)
0929 16:35:28.594776 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.594823 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.61:38969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:35:28.595034 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.61:38969: Network error: Client connection negotiation failed: client connection to 127.23.44.61:38969: connect: Connection refused (error 111)
I20260929 16:35:28.596419 13989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:40083,127.23.44.61:38969,127.23.44.60:39785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.61.keytab
--principal=kudu/127.23.44.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.61:38969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.44.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:28.597370 13989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:28.598275 13989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:35:28.606204 13994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:28.606279 13989 server_base.cc:1034] running on GCE node
W20260929 16:35:28.606182 13995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:28.606228 13997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:28.606868 13989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:28.607882 13989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:28.609064 13989 hybrid_clock.cc:648] HybridClock initialized: now 1790699728609040 us; error 46 us; skew 500 ppm
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:28.611696 13989 init.cc:377] Logged in from keytab as kudu/127.23.44.61@KRBTEST.COM (short username kudu)
I20260929 16:35:28.612459 13989 webserver.cc:460] Webserver started at http://127.23.44.61:44591/ using document root <none> and password file <none>
I20260929 16:35:28.612964 13989 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:28.613051 13989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:28.613301 13989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:28.615255 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/data/instance:
uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1"
format_stamp: "Formatted at 2026-09-29 16:35:28 on dist-test-slave-mlmz"
server_key: "58ebe61e887889d8af1ff16eefdf0f08"
server_key_iv: "960b0ab9ef6399dd9a2b8453786e582d"
server_key_version: "encryptionkey@0"
I20260929 16:35:28.615880 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/wal/instance:
uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1"
format_stamp: "Formatted at 2026-09-29 16:35:28 on dist-test-slave-mlmz"
server_key: "58ebe61e887889d8af1ff16eefdf0f08"
server_key_iv: "960b0ab9ef6399dd9a2b8453786e582d"
server_key_version: "encryptionkey@0"
I20260929 16:35:28.619263 13989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:35:28.621451 14004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:28.622515 13989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:28.622648 13989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/wal
uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1"
format_stamp: "Formatted at 2026-09-29 16:35:28 on dist-test-slave-mlmz"
server_key: "58ebe61e887889d8af1ff16eefdf0f08"
server_key_iv: "960b0ab9ef6399dd9a2b8453786e582d"
server_key_version: "encryptionkey@0"
I20260929 16:35:28.622746 13989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:28.640854 13989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:28.643431 13989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:28.643612 13989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:28.650068 13989 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.61:38969
I20260929 16:35:28.650389 14056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.61:38969 every 8 connection(s)
I20260929 16:35:28.651069 13989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/data/info.pb
I20260929 16:35:28.653602 14057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:28.657079 14057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:35:28.661293 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 13989
I20260929 16:35:28.661438 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-1/wal/instance
I20260929 16:35:28.661703 23728 external_mini_cluster.cc:1428] Setting key 72c1cc34a252a3f28535db44c5f52522
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.61@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.44.60@KRBTEST.COM; defaulting to no policy
I20260929 16:35:28.670141 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.658824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.23.44.62:40083)
0929 16:35:28.658960 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.658964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.659667 (+ 703us) server_negotiation.cc:407] Connection header received
0929 16:35:28.660560 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.660563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.660608 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.660679 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.661893 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.662702 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.663786 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.664017 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.664131 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.666490 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.666494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.666496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.666518 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.668038 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.668576 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.668581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.668586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.668676 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.669044 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.669048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.669051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.669238 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.669369 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.669855 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.669975 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:35:28.670830 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.659137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40083 (local address 127.0.0.1:42408)
0929 16:35:28.659554 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.659592 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:28.660377 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:28.660821 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:28.660827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:28.661161 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:28.661733 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.661746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.662819 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.662821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.663631 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.663639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.664163 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.664167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.664323 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.664910 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:28.664926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:28.666313 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:28.668185 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.668192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.668205 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.668465 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.668798 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.668803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.668805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.668940 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.669367 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:28.669374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:28.669609 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 16:35:28.669795 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 16:35:28.670083 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.44.60@KRBTEST.COM" created.
I20260929 16:35:28.672245 14057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.61@KRBTEST.COM for kudu/127.23.44.61@KRBTEST.COM
I20260929 16:35:28.681030 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.672733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:38969 (local address 127.0.0.1:54832)
0929 16:35:28.672917 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.672934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:28.673007 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:28.673527 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:28.673530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:28.673690 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:28.673769 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.673774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.674703 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.674706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.675226 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.675233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.675795 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.675799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.675890 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.676366 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:28.676377 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:28.677872 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:28.679690 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.679695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.679697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.679883 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.680163 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.680165 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.680166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.680269 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.680669 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:28.680671 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:28.680749 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 16:35:28.680830 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:35:28.680928 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 16:35:28.681324 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.672832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54832 (local address 127.23.44.61:38969)
0929 16:35:28.673156 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.673164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.673179 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:35:28.673208 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.673217 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.673415 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.673547 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.673874 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.674582 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.675337 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.675680 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.675793 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.678003 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.678011 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.678021 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.678041 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.679563 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.679984 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.679988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.679989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.680061 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.680355 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.680357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.680358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.680561 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.680659 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.680949 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.681109 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":239,"thread_start_us":157,"threads_started":1}
I20260929 16:35:28.682950 14057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:35:28.683624 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.683388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48756)
0929 16:35:28.683488 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.683560 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 16:35:28.683816 14007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
I20260929 16:35:28.684576 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.684359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48764)
0929 16:35:28.684466 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.684514 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:35:28.684703 14057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:39785: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
I20260929 16:35:28.686177 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.44.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.60.keytab.
Entry for principal kudu/127.23.44.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.60.keytab.
I20260929 16:35:28.687902 23728 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
/tmp/dist-test-taskNcOjI6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.60:39785
--webserver_interface=127.23.44.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.60.keytab
--principal=kudu/127.23.44.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.44.20:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.44.62:40083,127.23.44.61:38969,127.23.44.60:39785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.61@KRBTEST.COM
I20260929 16:35:28.695374 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.686688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.23.44.61:38969)
0929 16:35:28.686830 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.686835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.686849 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:35:28.686989 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.686992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.687037 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.687124 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.687728 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.688330 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.689805 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.690198 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.690306 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.692372 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.692380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.692382 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.692401 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.693858 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.694275 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.694280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.694282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.694352 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.694636 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.694641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.694642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.694794 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.694896 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.695144 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.695240 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:35:28.695401 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.686647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:38969 (local address 127.0.0.1:54834)
0929 16:35:28.686768 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.686783 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:28.686867 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:28.687158 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:28.687161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:28.687308 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:28.687445 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.687454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.688444 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.688451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.689672 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.689682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.690328 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.690335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.690481 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.690901 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:28.690921 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:28.692237 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:28.693983 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.693988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.693990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.694179 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.694460 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.694463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.694465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.694547 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.694888 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:28.694891 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:28.694994 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 16:35:28.695097 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 16:35:28.695227 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 16:35:28.696461 13982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:35:28.697149 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.696922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48780)
0929 16:35:28.697036 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.697087 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:35:28.697960 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.697665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48792)
0929 16:35:28.697865 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.697912 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260929 16:35:28.698160 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:39785: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
I20260929 16:35:28.729337 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } attempt: 1
I20260929 16:35:28.730152 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.729892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48808)
0929 16:35:28.730032 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.730085 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 16:35:28.730831 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.730635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48818)
0929 16:35:28.730737 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.730785 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 16:35:28.730969 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:39785: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
I20260929 16:35:28.733932 14057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } attempt: 1
I20260929 16:35:28.734522 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.734286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48822)
0929 16:35:28.734400 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.734465 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:35:28.735142 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.734959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48838)
0929 16:35:28.735054 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.735094 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 16:35:28.735321 14057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:39785: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
W20260929 16:35:28.793390 14064 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:35:28.793622 14064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:35:28.793679 14064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:35:28.793748 14064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:35:28.797611 14064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:35:28.797652 14064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:35:28.797670 14064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:35:28.797722 14064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:35:28.797755 14064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:35:28.802768 14064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.44.20:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.44.62:40083,127.23.44.61:38969,127.23.44.60:39785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/krb5kdc/kudu_127.23.44.60.keytab
--principal=kudu/127.23.44.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.44.60:39785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.44.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:35:28.803472 14057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } attempt: 2
I20260929 16:35:28.803679 14064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:35:28.804297 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.804012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48848)
0929 16:35:28.804171 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.804230 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:35:28.804751 14064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:35:28.804982 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.804766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48860)
0929 16:35:28.804875 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.804928 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:35:28.805128 14057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:39785: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
I20260929 16:35:28.806134 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } attempt: 2
I20260929 16:35:28.806752 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.806513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48872)
0929 16:35:28.806632 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.806691 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:35:28.807379 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.807178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48880)
0929 16:35:28.807277 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.807322 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 16:35:28.807518 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.44.60:39785: Network error: Client connection negotiation failed: client connection to 127.23.44.60:39785: connect: Connection refused (error 111)
W20260929 16:35:28.812778 14072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:28.812808 14069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:35:28.812804 14070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:35:28.812939 14064 server_base.cc:1034] running on GCE node
I20260929 16:35:28.813472 14064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:35:28.814469 14064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:35:28.815657 14064 hybrid_clock.cc:648] HybridClock initialized: now 1790699728815639 us; error 30 us; skew 500 ppm
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:35:28.818275 14064 init.cc:377] Logged in from keytab as kudu/127.23.44.60@KRBTEST.COM (short username kudu)
I20260929 16:35:28.819075 14064 webserver.cc:460] Webserver started at http://127.23.44.60:46773/ using document root <none> and password file <none>
I20260929 16:35:28.819581 14064 fs_manager.cc:359] Metadata directory not provided
I20260929 16:35:28.819664 14064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:35:28.819856 14064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:35:28.821750 14064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/data/instance:
uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78"
format_stamp: "Formatted at 2026-09-29 16:35:28 on dist-test-slave-mlmz"
server_key: "db909620849ffd66ecf994ff91411e28"
server_key_iv: "14528b073dcedf15e7880a2d8ffff3f7"
server_key_version: "encryptionkey@0"
I20260929 16:35:28.822270 14064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/wal/instance:
uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78"
format_stamp: "Formatted at 2026-09-29 16:35:28 on dist-test-slave-mlmz"
server_key: "db909620849ffd66ecf994ff91411e28"
server_key_iv: "14528b073dcedf15e7880a2d8ffff3f7"
server_key_version: "encryptionkey@0"
I20260929 16:35:28.825605 14064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:35:28.827795 14079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:35:28.828850 14064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:35:28.828981 14064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/data,/tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/wal
uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78"
format_stamp: "Formatted at 2026-09-29 16:35:28 on dist-test-slave-mlmz"
server_key: "db909620849ffd66ecf994ff91411e28"
server_key_iv: "14528b073dcedf15e7880a2d8ffff3f7"
server_key_version: "encryptionkey@0"
I20260929 16:35:28.829082 14064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:35:28.848549 14064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:35:28.851456 14064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:35:28.851642 14064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:35:28.858201 14064 rpc_server.cc:307] RPC server started. Bound to: 127.23.44.60:39785
I20260929 16:35:28.858470 14131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.44.60:39785 every 8 connection(s)
I20260929 16:35:28.859076 14064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/data/info.pb
I20260929 16:35:28.861590 14132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:35:28.863904 23728 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu as pid 14064
I20260929 16:35:28.864044 23728 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNcOjI6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699450427356-23728-0/minicluster-data/master-2/wal/instance
I20260929 16:35:28.864323 23728 external_mini_cluster.cc:1428] Setting key f1babc0aaeb5d74cc6d3bed5bb6b3402
I20260929 16:35:28.865566 14132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.60@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.60@KRBTEST.COM
I20260929 16:35:28.877717 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.867136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.23.44.62:40083)
0929 16:35:28.867261 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.867266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.867575 (+ 309us) server_negotiation.cc:407] Connection header received
0929 16:35:28.867809 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.867812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.867857 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.867925 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.868654 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.869451 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.870762 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.871106 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.871217 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.873933 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.873939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.873946 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.873993 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.875537 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.876079 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.876083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.876085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.876188 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.876516 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.876520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.876522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.876757 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.876865 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.877415 (+ 550us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.877540 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:35:28.878589 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.865596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.23.44.60:39785)
0929 16:35:28.865956 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.865980 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.866012 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:35:28.866627 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.866642 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.867053 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.867387 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.868224 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.869589 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.870683 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.871246 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.871389 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.874392 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.874400 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.874412 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.874442 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.876108 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.876529 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.876534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.876536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.876613 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.876876 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.876879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.876881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.877149 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.877390 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.877610 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.877874 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":235,"thread_start_us":119,"threads_started":1}
I20260929 16:35:28.878589 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.867067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.62:40083 (local address 127.0.0.1:42412)
0929 16:35:28.867444 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.867494 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:28.867671 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:28.868008 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:28.868013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:28.868377 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:28.868519 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.868529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.869546 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.869548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.870642 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.870654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.871263 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.871266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.871483 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.872124 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:28.872145 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:28.873776 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:28.875645 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.875655 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.875664 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.875973 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.876291 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.876293 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.876294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.876417 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.876879 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:28.876884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:28.877150 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 16:35:28.877360 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 16:35:28.877667 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":44,"spinlock_wait_cycles":2432,"thread_start_us":130,"threads_started":1}
I20260929 16:35:28.879769 14132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.60@KRBTEST.COM for kudu/127.23.44.61@KRBTEST.COM
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.62@KRBTEST.COM
I20260929 16:35:28.888736 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.880248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.61:38969 (local address 127.0.0.1:54846)
0929 16:35:28.880345 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.880359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:28.880445 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:28.880697 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:28.880700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:28.880844 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:28.880962 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.880969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.881879 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.881881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.882713 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.882724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.883207 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.883211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.883303 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.883848 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:28.883862 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:28.885162 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:28.887221 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.887225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.887227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.887436 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.887736 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.887741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.887743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.887838 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.888223 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:28.888226 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:28.888327 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 16:35:28.888440 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 16:35:28.888570 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 16:35:28.888754 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.880346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.23.44.61:38969)
0929 16:35:28.880481 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.880485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.880499 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:35:28.880543 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.880546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.880596 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.880670 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.881103 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.881769 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.882873 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.883100 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.883207 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.885330 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.885334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.885337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.885370 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.887090 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.887543 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.887548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.887550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.887639 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.887935 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.887940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.887942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.888121 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.888232 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.888497 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.888607 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:35:28.889417 14132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:35:28.889658 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.881020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.23.44.62:40083)
0929 16:35:28.881127 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.881132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.881143 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:35:28.881191 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.881194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.881232 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.881330 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.881981 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.882687 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.883557 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.883897 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.884013 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.886555 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.886558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.886560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.886580 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.887993 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.888387 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.888391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.888393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.888475 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.888738 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.888743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.888744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.888990 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.889175 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.889418 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.889523 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.60@KRBTEST.COM for kudu/127.23.44.60@KRBTEST.COM
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.44.61@KRBTEST.COM
I20260929 16:35:28.896308 14057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } attempt: 3
I20260929 16:35:28.898128 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.889802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48888)
0929 16:35:28.889889 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.889902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:28.889966 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:28.890332 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:28.890334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:28.890485 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:28.890555 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.890561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.891606 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.891609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.892219 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.892228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.892714 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.892717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.892793 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.893221 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:28.893234 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:28.894727 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:28.896420 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.896426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.896430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.896668 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.896994 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.896997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.897000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.897102 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.897611 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:28.897613 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:28.897689 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 16:35:28.897795 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 16:35:28.897914 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":1}
I20260929 16:35:28.898257 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.890017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.23.44.60:39785)
0929 16:35:28.890102 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.890106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.890117 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:35:28.890142 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.890145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.890180 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.890246 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.890678 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.891499 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.892334 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.892577 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.892704 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.894851 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.894854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.894856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.894876 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.896304 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.896757 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.896761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.896763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.896882 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.897208 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.897215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.897221 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.897479 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.897574 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.897884 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.898010 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":28}
I20260929 16:35:28.901036 14132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78: Bootstrap starting.
I20260929 16:35:28.902192 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.890872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54854 (local address 127.23.44.61:38969)
0929 16:35:28.890981 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.890985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.891025 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:35:28.891139 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.891141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.891170 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.891228 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.891964 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.892565 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.893456 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.893829 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.893907 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.896300 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.896303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.896306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.896331 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.898123 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.898602 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.898605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.898606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.898685 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.901319 (+ 2634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.901322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.901324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.901545 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.901838 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.901927 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.902032 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.61@KRBTEST.COM for kudu/127.23.44.60@KRBTEST.COM
I20260929 16:35:28.904080 14132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:28.905189 14132 log.cc:824] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:28.905830 23728 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:35:28.906683 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } attempt: 3
I20260929 16:35:28.907192 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.896718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48892)
0929 16:35:28.897145 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.897161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:28.897238 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:28.897517 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:28.897520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:28.897709 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:28.897812 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.897821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.899951 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.899954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.900696 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.900706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.901187 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.901191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.901325 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.902150 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:28.902164 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:28.903529 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:28.905608 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.905611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.905613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.905862 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.906162 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.906165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.906167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.906260 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.906747 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:28.906750 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:28.906849 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 16:35:28.906929 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:35:28.907059 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352}
I20260929 16:35:28.907704 14132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78: No bootstrap required, opened a new log
I20260929 16:35:28.907708 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.896875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.23.44.60:39785)
0929 16:35:28.897215 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.897228 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.897242 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:35:28.897353 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.897358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.897398 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.897504 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.898094 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.899836 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.900831 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.901055 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.901188 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.903689 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.903694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.903697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.903732 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.905488 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.905986 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.905989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.905990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.906070 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.906387 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.906391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.906393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.906624 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.907359 (+ 735us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.907435 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.907546 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":124,"threads_started":1}
I20260929 16:35:28.907048 13956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "042a8d36c5654fbea6b6c543838483fe"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42426
I20260929 16:35:28.911814 14057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1: Bootstrap starting.
Sep 29 16:35:28 dist-test-slave-mlmz krb5kdc[13901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.44.62@KRBTEST.COM for kudu/127.23.44.60@KRBTEST.COM
I20260929 16:35:28.911865 14132 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } }
I20260929 16:35:28.912210 14132 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdaf53f7cf284d9f9b6fcb0ae97efd78, State: Initialized, Role: FOLLOWER
I20260929 16:35:28.912555 14132 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } }
I20260929 16:35:28.913833 14132 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:28.914469 14057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:28.915294 14057 log.cc:824] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:28.915995 14139 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } } }
I20260929 16:35:28.916410 14139 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:28.916923 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.907037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.44.60:39785 (local address 127.0.0.1:48900)
0929 16:35:28.907140 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 16:35:28.907156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:35:28.907232 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:35:28.907600 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:35:28.907602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:35:28.907786 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:35:28.907908 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.907919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.908863 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.908956 (+ 93us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.909652 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:35:28.909662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.910265 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.910268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:35:28.910348 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.910814 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:35:28.910829 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:35:28.912214 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:35:28.914802 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.914806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.914808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.915060 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.915483 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:35:28.915499 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:35:28.915501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:35:28.915595 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:35:28.916469 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:35:28.916483 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:35:28.916571 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 16:35:28.916654 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:35:28.916771 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 16:35:28.917290 14057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1: No bootstrap required, opened a new log
I20260929 16:35:28.917550 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.907263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.23.44.60:39785)
0929 16:35:28.907376 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.907382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.907395 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:35:28.907426 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.907439 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.907474 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.908019 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.908095 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.908759 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.909773 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.910104 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.910242 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.912390 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.912394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.912396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.912442 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.914669 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.915183 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.915186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.915189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.915322 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.915768 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.915771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.915773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.916315 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.917220 (+ 905us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.917322 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.917424 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 16:35:28.919124 14151 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:35:28.919199 14151 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:35:28.920068 13982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe: Bootstrap starting.
I20260929 16:35:28.920145 14057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } }
I20260929 16:35:28.920282 14057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c8f26bfc3dd4ea3afab7d00f24721c1, State: Initialized, Role: FOLLOWER
I20260929 16:35:28.920499 14057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } }
I20260929 16:35:28.920857 14153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } } }
I20260929 16:35:28.920962 14153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:28.922020 13982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe: Neither blocks nor log segments found. Creating new log.
I20260929 16:35:28.922230 14057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:28.922801 13982 log.cc:824] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe: Log is configured to *not* fsync() on all Append() calls
I20260929 16:35:28.922251 14064 master_runner.cc:186] Error getting master registration for 127.23.44.62:40083: OK, instance_id { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" instance_seqno: 1790699728443942 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:35:28.925782 14064 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:35:28.926115 13982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe: No bootstrap required, opened a new log
I20260929 16:35:28.928817 13982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } }
I20260929 16:35:28.925318 13989 master_runner.cc:186] Error getting master registration for 127.23.44.62:40083: OK, instance_id { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" instance_seqno: 1790699728443942 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:35:28.929005 13982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042a8d36c5654fbea6b6c543838483fe, State: Initialized, Role: FOLLOWER
I20260929 16:35:28.929102 13989 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:35:28.929313 13982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } }
I20260929 16:35:28.929785 14166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } } }
I20260929 16:35:28.930011 14166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:28.930523 13982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:35:28.930610 13956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:35:28.930696 13956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [term 0 FOLLOWER]: Advancing to term 1
W20260929 16:35:28.931807 14165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:35:28.931864 14165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:35:28.931964 13956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } }
I20260929 16:35:28.933357 14031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "042a8d36c5654fbea6b6c543838483fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1"
I20260929 16:35:28.933462 14106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "042a8d36c5654fbea6b6c543838483fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78"
I20260929 16:35:28.933565 14031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:28.933688 14106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:35:28.934798 14031 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042a8d36c5654fbea6b6c543838483fe in term 1.
I20260929 16:35:28.934921 14106 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042a8d36c5654fbea6b6c543838483fe in term 1.
I20260929 16:35:28.935369 13932 leader_election.cc:302] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042a8d36c5654fbea6b6c543838483fe, bdaf53f7cf284d9f9b6fcb0ae97efd78; no voters:
I20260929 16:35:28.935637 14166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:35:28.932893 13956 leader_election.cc:288] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [CANDIDATE]: Term 1 election: Requested vote from peers 9c8f26bfc3dd4ea3afab7d00f24721c1 (127.23.44.61:38969), bdaf53f7cf284d9f9b6fcb0ae97efd78 (127.23.44.60:39785)
I20260929 16:35:28.935818 14166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [term 1 LEADER]: Becoming Leader. State: Replica: 042a8d36c5654fbea6b6c543838483fe, State: Running, Role: LEADER
I20260929 16:35:28.936112 14166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } }
I20260929 16:35:28.937700 14166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 042a8d36c5654fbea6b6c543838483fe. Latest consensus state: current_term: 1 leader_uuid: "042a8d36c5654fbea6b6c543838483fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } } }
I20260929 16:35:28.937803 14166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [sys.catalog]: This master's current role is: LEADER
W20260929 16:35:28.941208 13935 tablet.cc:2369] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:35:28.946602 14178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:35:28.946681 14178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:35:28.950551 14185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:35:28.951548 14185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:35:28.956146 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.939101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.23.44.62:40083)
0929 16:35:28.939200 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.939204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.940350 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 16:35:28.940887 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.940890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.940935 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.940983 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.942783 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.943553 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.946747 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.947073 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.947151 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.950941 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.950945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.950948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.950973 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.953182 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.954157 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.954161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.954163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.954233 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.954669 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.954672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.954674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.954878 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.954983 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.955888 (+ 905us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.956001 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 16:35:28.957114 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.938051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.23.44.60:39785)
0929 16:35:28.938151 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.938156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.939876 (+ 1720us) server_negotiation.cc:407] Connection header received
0929 16:35:28.940658 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.940662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.940711 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.940760 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.944814 (+ 4054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.945594 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.947656 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.947991 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.948065 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.950566 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.950570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.950573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.950598 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.952434 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.953968 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.953972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.953974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.954052 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.955253 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.955257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.955258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.955457 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.955561 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.956854 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.956973 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 16:35:28.962662 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:28.938824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.23.44.61:38969)
0929 16:35:28.938925 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:28.938939 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:28.941302 (+ 2363us) server_negotiation.cc:407] Connection header received
0929 16:35:28.941344 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:28.941347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:28.941388 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:28.941464 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:28.947565 (+ 6101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.948359 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.949526 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:28.949858 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:28.949932 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:28.953390 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:35:28.953394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:35:28.953397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:35:28.953424 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:35:28.955089 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.961397 (+ 6308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.961401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.961403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.961489 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:35:28.961778 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:35:28.961781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:35:28.961783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:35:28.961986 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:35:28.962099 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:28.962412 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 16:35:28.962520 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 16:35:28.966111 14106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [term 1 FOLLOWER]: Refusing update from remote peer 042a8d36c5654fbea6b6c543838483fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:28.966698 14166 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:28.972684 14139 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 042a8d36c5654fbea6b6c543838483fe. Latest consensus state: current_term: 1 leader_uuid: "042a8d36c5654fbea6b6c543838483fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } } }
I20260929 16:35:28.972812 14139 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:28.974387 14031 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [term 1 FOLLOWER]: Refusing update from remote peer 042a8d36c5654fbea6b6c543838483fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:35:28.974987 14166 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:35:28.978534 14153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 042a8d36c5654fbea6b6c543838483fe. Latest consensus state: current_term: 1 leader_uuid: "042a8d36c5654fbea6b6c543838483fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } } }
I20260929 16:35:28.978649 14153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:28.984611 14179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "042a8d36c5654fbea6b6c543838483fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } } }
I20260929 16:35:28.984737 14179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:28.985841 14153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "042a8d36c5654fbea6b6c543838483fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } } }
I20260929 16:35:28.985941 14153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:28.986279 14190 mvcc.cc:205] Tried to move back new op lower bound from 7334706089839149056 to 7334706089725272064. Current Snapshot: MvccSnapshot[applied={T|T < 7334706089839149056}]
I20260929 16:35:28.989048 14193 mvcc.cc:205] Tried to move back new op lower bound from 7334706089839149056 to 7334706089725272064. Current Snapshot: MvccSnapshot[applied={T|T < 7334706089839149056}]
I20260929 16:35:28.991822 14166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "042a8d36c5654fbea6b6c543838483fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } } }
I20260929 16:35:28.991930 14166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe [sys.catalog]: This master's current role is: LEADER
I20260929 16:35:28.992846 14139 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "042a8d36c5654fbea6b6c543838483fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042a8d36c5654fbea6b6c543838483fe" member_type: VOTER last_known_addr { host: "127.23.44.62" port: 40083 } } peers { permanent_uuid: "9c8f26bfc3dd4ea3afab7d00f24721c1" member_type: VOTER last_known_addr { host: "127.23.44.61" port: 38969 } } peers { permanent_uuid: "bdaf53f7cf284d9f9b6fcb0ae97efd78" member_type: VOTER last_known_addr { host: "127.23.44.60" port: 39785 } } }
I20260929 16:35:28.992950 14139 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:35:28.995126 14191 mvcc.cc:205] Tried to move back new op lower bound from 7334706089839149056 to 7334706089725272064. Current Snapshot: MvccSnapshot[applied={T|T < 7334706089839149056}]
I20260929 16:35:28.998682 14185 catalog_manager.cc:1347] Generated new cluster ID: eda4163ac5b84fd5876e112080e1f9c3
I20260929 16:35:28.998745 14185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:35:29.021328 14185 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:35:29.022478 14185 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:35:29.046161 14185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 042a8d36c5654fbea6b6c543838483fe: Generated new TSK 0
I20260929 16:35:29.047004 14185 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:35:29.076403 14135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.073465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.23.44.60:39785)
0929 16:35:29.073603 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.073607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.075330 (+ 1723us) server_negotiation.cc:407] Connection header received
0929 16:35:29.075540 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.075543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.075594 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.075649 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.076284 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48922: BlockingRecv error: recv got EOF from 127.0.0.1:48922 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 16:35:29.077771 13986 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.074079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.23.44.62:40083)
0929 16:35:29.074199 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.074204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.076868 (+ 2664us) server_negotiation.cc:407] Connection header received
0929 16:35:29.077036 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.077039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.077088 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.077139 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.077657 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42434: BlockingRecv error: recv got EOF from 127.0.0.1:42434 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 16:35:29.079121 14062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.074752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.23.44.61:38969)
0929 16:35:29.074861 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.074867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.078200 (+ 3333us) server_negotiation.cc:407] Connection header received
0929 16:35:29.078437 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.078440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.078489 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.078542 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.079014 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54872: BlockingRecv error: recv got EOF from 127.0.0.1:54872 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 16:35:29.083366 14062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.081954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.23.44.61:38969)
0929 16:35:29.082059 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.082062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.082490 (+ 428us) server_negotiation.cc:407] Connection header received
0929 16:35:29.082691 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.082695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.082760 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.082812 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.083281 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54886: BlockingRecv error: recv got EOF from 127.0.0.1:54886 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 16:35:29.088204 14062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.086823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54900 (local address 127.23.44.61:38969)
0929 16:35:29.086918 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.086922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.087355 (+ 433us) server_negotiation.cc:407] Connection header received
0929 16:35:29.087554 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.087557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.087604 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.087653 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.088120 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54900: BlockingRecv error: recv got EOF from 127.0.0.1:54900 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 16:35:29.093649 14062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.092192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.23.44.61:38969)
0929 16:35:29.092281 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.092285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.092720 (+ 435us) server_negotiation.cc:407] Connection header received
0929 16:35:29.092902 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.092905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.092948 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.092994 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.093568 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54910: BlockingRecv error: recv got EOF from 127.0.0.1:54910 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 16:35:29.101019 14062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.099666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.23.44.61:38969)
0929 16:35:29.099785 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.099789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.100208 (+ 419us) server_negotiation.cc:407] Connection header received
0929 16:35:29.100376 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.100379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.100436 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.100483 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.100937 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54924: BlockingRecv error: recv got EOF from 127.0.0.1:54924 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 16:35:29.112429 14062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.111001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54938 (local address 127.23.44.61:38969)
0929 16:35:29.111094 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.111098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.111533 (+ 435us) server_negotiation.cc:407] Connection header received
0929 16:35:29.111694 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.111697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.111740 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.111786 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.112298 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54938: BlockingRecv error: recv got EOF from 127.0.0.1:54938 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 16:35:29.131762 14062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.130554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.23.44.61:38969)
0929 16:35:29.130662 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.130666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.130895 (+ 229us) server_negotiation.cc:407] Connection header received
0929 16:35:29.131047 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.131051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.131149 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.131233 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.131629 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54948: BlockingRecv error: recv got EOF from 127.0.0.1:54948 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 16:35:29.166900 14062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.165734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54950 (local address 127.23.44.61:38969)
0929 16:35:29.165886 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.165891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.166059 (+ 168us) server_negotiation.cc:407] Connection header received
0929 16:35:29.166220 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.166223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.166279 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.166376 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.166796 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54950: BlockingRecv error: recv got EOF from 127.0.0.1:54950 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 16:35:29.234308 14062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.232988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.23.44.61:38969)
0929 16:35:29.233212 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.233216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.233408 (+ 192us) server_negotiation.cc:407] Connection header received
0929 16:35:29.233569 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.233571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.233626 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.233702 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.234185 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54958: BlockingRecv error: recv got EOF from 127.0.0.1:54958 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260929 16:35:29.365360 14062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.364278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.23.44.61:38969)
0929 16:35:29.364462 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.364466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.364500 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:35:29.364635 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.364637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.364686 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.364769 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.365215 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54964: BlockingRecv error: recv got EOF from 127.0.0.1:54964 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 16:35:29.624842 14062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:35:29.623587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.23.44.61:38969)
0929 16:35:29.623721 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:29.623728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:29.623804 (+ 76us) server_negotiation.cc:407] Connection header received
0929 16:35:29.623992 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:29.623996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:29.624055 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:29.624143 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:35:29.624715 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54976: BlockingRecv error: recv got EOF from 127.0.0.1:54976 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:35:29.920400 14151 catalog_manager.cc:1259] Loaded cluster ID: eda4163ac5b84fd5876e112080e1f9c3
I20260929 16:35:29.920482 14151 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78: loading cluster ID for follower catalog manager: success
I20260929 16:35:29.922309 14151 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78: acquiring CA information for follower catalog manager: success
I20260929 16:35:29.923520 14151 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bdaf53f7cf284d9f9b6fcb0ae97efd78: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:35:29.933012 14165 catalog_manager.cc:1259] Loaded cluster ID: eda4163ac5b84fd5876e112080e1f9c3
I20260929 16:35:29.933238 14165 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1: loading cluster ID for follower catalog manager: success
I20260929 16:35:29.934757 14165 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1: acquiring CA information for follower catalog manager: success
I20260929 16:35:29.935782 14165 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c8f26bfc3dd4ea3afab7d00f24721c1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:35:30.142701 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:30.138849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.23.44.61:38969)
0929 16:35:30.139108 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:30.139113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:30.139128 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:35:30.139281 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:30.139284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:30.139420 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:30.139541 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:30.139801 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.140330 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.141180 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.141429 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.141541 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:30.141877 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:30.142286 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:30.142421 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:30.142483 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 16:35:30.142539 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":67,"threads_started":1}
I20260929 16:35:30.149113 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:30.145794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.23.44.60:39785)
0929 16:35:30.146031 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:30.146034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:30.146048 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:35:30.146184 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:30.146186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:30.146288 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:30.146385 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:30.146587 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.147100 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.147907 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.148126 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.148218 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:30.148447 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:30.148806 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:30.148895 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:30.148959 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 16:35:30.149004 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260929 16:35:30.155856 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:30.152373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.23.44.60:39785)
0929 16:35:30.152715 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:30.152721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:30.152735 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:35:30.152780 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:30.152785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:30.152834 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:30.152905 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:30.153129 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.153713 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.154708 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.155042 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.155142 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:30.155380 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:30.155448 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:30.155514 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:30.155660 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 16:35:30.155719 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260929 16:35:30.156036 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:30.151876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.23.44.61:38969)
0929 16:35:30.152010 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:30.152014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:30.152356 (+ 342us) server_negotiation.cc:407] Connection header received
0929 16:35:30.152439 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:30.152442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:30.152492 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:30.152577 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:30.152839 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.153587 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.154807 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.155134 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.155219 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:30.155572 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:30.155647 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:30.155690 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:30.155844 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 16:35:30.155889 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:35:30.156661 14273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:35:30.151843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.23.44.62:40083)
0929 16:35:30.152097 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 16:35:30.152102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:35:30.152430 (+ 328us) server_negotiation.cc:407] Connection header received
0929 16:35:30.152524 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:35:30.152528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:35:30.152674 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:35:30.152788 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:35:30.153170 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.153933 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.155024 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:35:30.155374 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:35:30.155491 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:35:30.155889 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:35:30.156179 (+ 290us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:35:30.156269 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 16:35:30.156418 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 16:35:30.156480 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260929 16:35:30.159478 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 13914
I20260929 16:35:30.165990 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 13989
I20260929 16:35:30.171464 23728 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNcOjI6/build/debug/bin/kudu with pid 14064
2026-09-29T16:35:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3065 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6064 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279739 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST